Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add json schema support #231

Merged
merged 7 commits into from
Jul 17, 2021

Conversation

bakjos
Copy link
Contributor

@bakjos bakjos commented Jul 7, 2021

No description provided.

Copy link
Contributor

@weeco weeco left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey,
thanks for tackling this feature. I left a few comments.

Have you tested the changes against a schema registry / topic that uses a JSON schema?

backend/pkg/kafka/deserializer.go Outdated Show resolved Hide resolved
frontend/src/state/restInterfaces.ts Outdated Show resolved Hide resolved
@@ -741,7 +741,12 @@ const apiStore = {
const rq = cachedApiRequest(`./api/schemas/subjects/${subjectName}/versions/${version}`, force) as Promise<SchemaDetailsResponse>;

return rq
.then(({ schemaDetails }) => (this.schemaDetails = schemaDetails))
.then(({ schemaDetails }) => {
if ( schemaDetails && typeof schemaDetails.schema === "string" && schemaDetails.type !== "PROTOBUF") {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the typeof check is not really needed as we rely on the correctness of our typescript types, but @rikimaru0345 might chime in here as he's mostly responsible for the frontend code.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@weeco The main reason I did this was that even for Avro schemas it was not showing the detail in the schema details page, and the issue was because the schema object is sent as string from the backend, adding this validation/conversion solved the issue and showed the schema correctly,

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no, the typeof check is fine.
yes, it is assumed that the backend currently sends schema as a string and not anything else, but defensive programming is always a good idea.

@bakjos
Copy link
Contributor Author

bakjos commented Jul 7, 2021

@weeco Thanks for all the comments, Yes I tested it against the three types of schemas (JSON, PROTOBUF, and AVRO)

@bakjos bakjos requested a review from weeco July 7, 2021 16:58
@rikimaru0345
Copy link
Contributor

Code looks good to me.
@bakjos I'll check it out and test it later today or tomorrow.
If I read it correctly this PR will make it so the schema details page will now show the schema for Avro types, right?

@bakjos
Copy link
Contributor Author

bakjos commented Jul 12, 2021

@rikimaru0345 yes, it should show the fields for both Avro and Json schemas

Comment on lines +140 to +144
if d.SchemaService != nil && len(payload) > 5 && payload[0] == byte(0) {
schemaID := binary.BigEndian.Uint32(payload[1:5])
trimmed := bytes.TrimLeft(payload[5:], " \t\r\n")
startsWithJSON := trimmed[0] == '[' || trimmed[0] == '{'
if startsWithJSON {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@weeco from my comment in the PR thread: see here

@rikimaru0345
Copy link
Contributor

@weeco
Tested the PR, works fine for me.
As far as frontend changes go, I looked through all of them and they look good to me! 😄

I've also looked through the backend changes and the only thing I noticed is this: https://github.com/cloudhut/kowl/pull/231/files?diff=split&w=1#r671009018

Do you think that is "safe" enough for the detection?
I guess a payload starting with a zero byte + schema ID + json should be pretty obvious and robust, right?

I mean, I'd even go as far as saying that - compared to our other "auto detect payload type" checks - this is one of the more robust checks we have in there 😂. So I doubt there's much (anything really) that can be improved there.

In any case, the other backend changes seem fine to me as well (though I didn't check them as deeply as the frontend).

From my side this PR looks perfectly fine! 👍
So when you've looked at the backend stuff feel free to press that merge button!

@weeco weeco merged commit 578511e into redpanda-data:master Jul 17, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants