New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Decimal field with default value causes exception during deserialization #833
Comments
I'm having the same issue with |
There is also problem with other logical types. |
@gunnarmorling did you get a solution to the above on how to treat decimal defaults values |
Nope, no solution really apart from using JSON instead of Avro :( |
Can i change AvroData from schema-registry to fix all logicalType? |
Indeed we are encountering the same problem as well. Which was generated from: Hits the error when deserializing he message through a KC sink: |
Hey @rhauch, is there any chance to bump up this issue in terms of priority? We've received a few reports of Debezium users about it; unfortunately there isn't really anything we can do from the connector side (apart from disabling to propagate the default values to the schema, which might be a connector option we could add). Thanks a lot! |
Hello Everyone, We at Shutterstock were having the same issue with Debezium data and Decimal Logical Types with default Values. I created a fix for the problem and have tested and verified it in our CF4 cluster. The issue boils down to the fact that Avro Decimal Logical types store values as a Byte Arrays. Debezium sets the Default Value as Base64 encoded Byte Arrays and record values as Big Integer Byte Arrays The fix requires making changes in both io.confluent.connect.avro.AvroData and org.apache.kafka.connect.data.ConnectSchema to allow the Byte Class type in the validation hash maps. I've created a JIRA ticket for this in Apache Kafka: https://issues.apache.org/jira/browse/KAFKA-7688 If I can get GitHub permissions for both this repo and in Apache Kafka I can create a branch and a PR with my changes Thanks! |
@TheGreatAbyss , simply fork this repository and create a PR the normal way (pushing to your fork). There are no privileges required to create a PR with a suggested change. |
Hi @rhauch Here is my PR in my own Git Repo: TheGreatAbyss/schema-registry#1. This combined with a similar minor change in org.apache.kafka.connect.data.ConnectSchema.java fixes the problem for Debezium. |
@gunnarmorling, unfortunately, we're already frozen for the 5.1.1 release. I'm trying to determine whether we can still get this fix in. When fixing #985 I did find and fix a problem when the default values were set using a ByteBuffer (rather than It looks like #1014 may be the correct solution, but it's also not quite correct since it's using Avro decimal. |
@gunnarmorling I've done more research, and have confirmed there are problems with default values for any logical type (e.g., Decimal, Time, Date, and Timestamp) and another issue with BYTES; see also #983, #695. I also have a proposed fix (#1020) for the 5.1.x branch (trying to get it into 5.1.1 but might be too late as we are currently in freeze for that release). |
Thanks for confirming, @rhauch. If that doesn't get into 5.1.1, what would be the next release window then? If it helps to make a decision, I can confirm that this has been reported by multiple users of Debezium and unfortunately there isn't a workaround (unless using JSON of course). We've been thinking about offering an option for disabling the export of default values as a temporary measure, but it would be nice of course if this could be avoided. Thanks again for any efforts on this issue, that's much appreciated! |
@gunnarmorling, 5.1.2 has just been released, and this fix is in that version: https://github.com/confluentinc/schema-registry/commits/v5.1.2 |
Ah, that's excellent news; thanks for the heads-up!
… |
In my Kafka Connect schema I have a field of type
Decimal
with a default value of1.23
. This value and schema can be serialized using the Avro serializer, but during deserialization (within a KC sink connector), this causes an exception:From what I can say, the reason seems to be that the default value as retrieved from the Avro message (which is a byte array containing a single byte 123) is passed as is as the default value to the schema builder of the Connect schema. There it only allows
BigDecimal
as default values, though. Interestingly, this apparently worked with 4.0.0, but I'm seeing this exception as of 4.1.0.The text was updated successfully, but these errors were encountered: