refactor deserialization to be safer #82
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
isterin
reviewed
Aug 6, 2023
| @@ -104,6 +104,7 @@ object AvroUtils extends LazyLogging { | |||
| headers | |||
| ) | |||
| ) | |||
| } | |||
|
|
|||
| implicit class RichGenericRecord(genericRecord: GenericRecord) { | |||
There was a problem hiding this comment.
I think you can replace the whole toSpecific method with this:
def toSpecific[A <: GenericRecord](instance: A): A =
SpecificData.get().deepCopy(instance.getSchema, genericRecord).asInstanceOf[A]
There was a problem hiding this comment.
I tested it briefly and it seems to work
There was a problem hiding this comment.
Seems to work meaning, the way it's working now. I do have an issue though with logical type conversions. Still trying to figure it out. It's not an issue with your changes. I think it's an avrohugger limitation and/or something else I'm missing. I'll explain in slack.
There was a problem hiding this comment.
That's brilliant! Good find.
There was a problem hiding this comment.
Given that this can fail, we should wrap it in a Try, no?
nextdude
force-pushed
the
refactor-deserialization-again
branch
from
6c16b49
to
246638b
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Several changes to deserialization to make it more robust
ConfluentAvroRegistryKafkaRecordDeserializationSchemanow respects thespecific.avro.readerconfig. If you do not specify a setting, one will be selected based on whether the embedded avro record class is Generic or Specific. If you specifyspecific.avro.reader = falseand the embedded avro record class is Specific, the schema will deserialize as a Generic record, then attempt to convert to a Specific record instance.AvroUtils.toEmbeddedAvroInstance) is now wrapped in aTry, as it may fail.AvroUtils.toSpecificutility method implementation with call toSpecificData.deepCopy().GlueAvroRegistryKafkaRecordDeserializationSchemahas a smarter key deserializer that will fallback to Confluent if you've configured a confluent fallback (so that string keys are properly processed).