Support SchemaRegistry alternative serialization format

[npepinpe]

Add support to specify configuration for Confluent's SchemaRegistry so users can use their own schemas for serialization/deserialization.

[alex-basiuk]

Is there any plans to implement this feature?

[npepinpe]

Hi, I haven't really had time lately to work on this project, unfortunately, and probably won't in the near future. I'd be happy to work with you or anyone on this though.

[alex-basiuk]

Hi, I haven't really had time lately to work on this project, unfortunately, and probably won't in the near future. I'd be happy to work with you or anyone on this though.

My java is a bit rusty but I'll try.

[npepinpe]

Sorry, I guess I wasn't very clear 😅 I can gladly do the Java part, that's probably the smallest part. I'm just not that familiar with SchemaRegistry beyond what it's for, so researching what would need to be done, how we could do it, what's the best approach, etc., that's the part I don't have time for unfortunately.

That said, I'm working on 2.0 for this exporter and I'm thinking of ditching the protobuf stuff entirely for JSON. Maybe that already helps solve any issue you have? I think with Kafka Connect it's pretty easy later to map JSON stuff to a different schema all through connect, so maybe that would be alright? Again I don't have a lot of production experience with that though.

[alex-basiuk]

Ideally, it should be up to the user to choose a serialization format. But JSON is certainly "friendlier" and would solve my issue.

[npepinpe]

I could probably manage something with Jackson for this release where you can pick something provided by Jackson, e.g. JSON, YAML, CBOR, etc. My issue with Schema Registry is mostly that I lack knowledge to make an informed decision - from what I got, I'd have to define and maintain a schema for the records? Or is it possible to delegate this task to the user and pull the schema from Schema registry? I'd still have to map it manually to it, which means any changes to the protocol is a change I have to do here. But maybe I misunderstood how it's meant to be used.

It's probably a good feature in the long run anyway, but considering how much time I have atm for this maybe not ideal 😅

[npepinpe]

Hey @alex-basiuk, I just released 2.0.0 of the exporter, which switches to JSON, so that might already help you out.

I started looking into support for Schema Registry, and I think I'm getting some idea of how the design could look like. It'll be the next feature I work on for sure once I have a more solid idea of how the integration can look like, mostly from a configuration point of view.

I'll update then