Decouple avro reader and writer serdes #157
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I believe this resolves a few issues including #152, #137. It also represents part of the work required to resolve #136 although that might need a bit more work on the serialization side and could certainly use a test and documentation to demonstrate it's use. I plan to follow up on this with another guide in the docs section all about configuring the various kafka client entities (e.g. producers, consumers, serdes etc).
The main change is to add a
:deserialization-properties
optional keyword to the confluent serde constructor. Including the property "specific.avro.reader" in these properties means that any schema supplied to the deserializer will be used in combination with the writer's schema to decode the message. The reader schema must be compatible with the writer's schema in order for this to work correctly. See the schema resolution rules in the related links for the details about how this works.Another change is that it is no longer an error if you try to create an avro serde without specifying a local copy of the schema. This can be used to consume data produced by some other process that does have access to the schema or which generates it's own schema on the fly (e.g. kafka-connect, debezium etc).
Finally I tried to remove some of the magic from the serde-resolver function. It should remain logically the same (except for not throwing an error as described above) but it is hopefully a bit clearer what's happening now.
Related Links: