Schema gets updated with every message produce

[saad1200]

I have notices an issue with produce with configuration with schema and schema serializer. For some reason the defaults (which can be null) are removed from from the server schema as soon as I run the tests.
It seems the goavro lib that is being used does this as part of serialization. The problem is, this schema later is used as in a post request to fitch the schema id from the server. This post request update the server schema in this case. I could not find a way that we provide the schema id to bypass this logic/behavior.

[ssam5532]

Any update on this?

[mostafa]

Hey @saad1200 @ssam5532!

Can any of you provide an example script (and/or configuration) you use, so I can test it? Or at least explain more details.

[saad1200]

@mostafa Any produce code example from the repo can be used to reproduce this. It has been few months now, but I think it is an issue with goavro lib.

we noticed that xk6-kafka is posting a new schema when called with produceWithConfiguration. Looks like it’s because they’re stripping out some of the java specific types and before posting the schema to the registry (which I guess they need to do, to get an ID).

xk6-kafka/schemaRegistry.go

Line 59 in 57dfc78

req, err := http.NewRequest("POST", url, bytes.NewReader([]byte(body)))

If you look at line 54 and 56, goavro’s codec is turning the schema into and object, then making it into a string again on line 56.

[saad1200]

@ssam5532 our work around is to setup wiremock and use it as schema registry proxy. It is another dependency and far from ideal solution but allowed us to continue using K6.

[mostafa]

@saad1200
I suppose this can be fixed by trying to fetch the schema first before posting one. We can also use something like srclient and have it handle everything, but it's an extra dependency for just two HTTP requests.

Update:
Upon further investigation, I found out that it uses the internal HTTP client of Go, goavro v2, and jsonschema v5 as dependencies, so it's okay for our use-case. I'll try to see how I can integrate it and make a PR when ready.

[mostafa]

@saad1200 @ssam5532 I'd be happy to have your feedback on #52.

[mostafa]

@saad1200 @ssam5532
The PR (#52) is merged now, and you can test it to see if it fixes the issue or not. I haven't released a version yet, so you need to build the latest main according to xk6 build instructions to test it. Let me know if you face any issues along the way!
Also, based on my tests, the schema doesn't get updated.

[saad1200]

@mostafa sorry for the delay. I will block some time next weekend to test the changes.

[kusumkappdirect]

@saad1200 , can you please help me here. , my schema also get updated with each sent message on topic , how you retrieve the schema first and then send same ?

[saad1200]

@kusumkappdirect
If you are using latest release and still having this issue all you need to do is to setup wiremock on the dev server or locally, then in the k6 setup method get the schema from the schema registry and stub the methods to return that schema using wiremock. Pass wiremock address instead of the schema registry to the k6 extension produce method.
I will share my code with you soon