[Issue #10427] Add AvroSchema UUID support fix #10428
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
vroyer
force-pushed
the
fix/avro_schema_uuid
branch
2 times, most recently
from
b5ec4f4
to
fbe13be
Compare
eolivelli
requested changes
Apr 29, 2021
pulsar-client/src/test/java/org/apache/pulsar/client/impl/schema/AvroSchemaTest.java
Show resolved
Hide resolved
pulsar-client/src/test/java/org/apache/pulsar/client/impl/schema/AvroSchemaTest.java
Outdated
Show resolved
Hide resolved
vroyer
force-pushed
the
fix/avro_schema_uuid
branch
from
ea763d0
to
27cfead
Compare
eolivelli
requested changes
Apr 29, 2021
pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/AvroSchema.java
Outdated
Show resolved
Hide resolved
|
Because the issue happens when creating the AVRO schema from a POJO class. Maybe a java annotation on BigDecimal fields could provide the scale ?
… Le 30 avr. 2021 à 15:38, Enrico Olivelli ***@***.***> a écrit :
@eolivelli commented on this pull request.
In pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/AvroSchema.java:
> @@ -127,7 +127,12 @@ public static void addLogicalTypeConversions(ReflectData reflectData, boolean js
}
}
reflectData.addLogicalTypeConversion(new Conversions.UUIDConversion());
- reflectData.addLogicalTypeConversion(new Conversions.DecimalConversion());
+ reflectData.addLogicalTypeConversion(new Conversions.DecimalConversion() {
+ @OverRide
+ public org.apache.avro.Schema getRecommendedSchema() {
+ return LogicalTypes.decimal(100,51).addToSchema(org.apache.avro.Schema.create(org.apache.avro.Schema.Type.BYTES));
we already have "isJsr310ConversionEnabled" and "getAlwaysAllowNull"
why don't we store precision and scale the same way ? this way they will be stored in the registry and consumers will use the right version of the schema
—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub, or unsubscribe.
|
vroyer
force-pushed
the
fix/avro_schema_uuid
branch
from
a803d21
to
9169403
Compare
eolivelli
requested changes
May 3, 2021
pulsar-client/src/test/java/org/apache/pulsar/client/impl/schema/AvroSchemaTest.java
Outdated
Show resolved
Hide resolved
vroyer
force-pushed
the
fix/avro_schema_uuid
branch
2 times, most recently
from
caa75c1
to
f61e75b
Compare
sijie
requested changes
May 3, 2021
pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/util/SchemaUtil.java
Show resolved
Hide resolved
eolivelli
approved these changes
May 20, 2021
pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/util/SchemaUtil.java
Show resolved
Hide resolved
vroyer
force-pushed
the
fix/avro_schema_uuid
branch
from
e17d00a
to
29113ee
Compare
vroyer
force-pushed
the
fix/avro_schema_uuid
branch
from
29113ee
to
8483966
Compare
|
@sijie I believe that now the patch is really good to go. please take another look when you have time |
sijie
approved these changes
Jul 12, 2021
eolivelli
pushed a commit
that referenced
this pull request
Jul 13, 2021
hangc0276
pushed a commit
that referenced
this pull request
Jan 29, 2022
### Motivation I can use Avro schema with BigDecimal in Pulsar 2.8.0, but this doesn't work on Pulsar 2.8.1, so I check this codebase and PR, found #10428 breaks this. The following is error log: ``` org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154) ~[spring-beans-5.3.4.jar:5.3.4] ... 34 common frames omitted Caused by: java.lang.UnsupportedOperationException: No recommended schema for decimal (scale is required) at org.apache.pulsar.shade.org.apache.avro.Conversions$DecimalConversion.getRecommendedSchema(Conversions.java:73) ~[pulsar-client-2.8.1.jar:2.8.1] at org.apache.pulsar.shade.org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:696) ~[pulsar-client-2.8.1.jar:2.8.1] at org.apache.pulsar.shade.org.apache.avro.reflect.ReflectData.createFieldSchema(ReflectData.java:873) ~[pulsar-client-2.8.1.jar:2.8.1] at org.apache.pulsar.shade.org.apache.avro.reflect.ReflectData$AllowNull.createFieldSchema(ReflectData.java:92) ~[pulsar-client-2.8.1.jar:2.8.1] at org.apache.pulsar.shade.org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:736) ~[pulsar-client-2.8.1.jar:2.8.1] at org.apache.pulsar.shade.org.apache.avro.specific.SpecificData$3.computeValue(SpecificData.java:328) ~[pulsar-client-2.8.1.jar:2.8.1] at org.apache.pulsar.shade.org.apache.avro.specific.SpecificData$3.computeValue(SpecificData.java:325) ~[pulsar-client-2.8.1.jar:2.8.1] at java.base/java.lang.ClassValue.getFromHashMap(ClassValue.java:228) ~[na:na] at java.base/java.lang.ClassValue.getFromBackup(ClassValue.java:210) ~[na:na] at java.base/java.lang.ClassValue.get(ClassValue.java:116) ~[na:na] at org.apache.pulsar.shade.org.apache.avro.specific.SpecificData.getSchema(SpecificData.java:339) ~[pulsar-client-2.8.1.jar:2.8.1] ... 52 common frames omitted ``` I think that Avro cannot work on using the ReflectData with Conversions.DecimalConversion to parse the BigDecimal field without AvroSchema when getting schema, but the Avro ReflectDatumWriter and ReflectDatumReader are working, it seems that Avro support for BigDecimal is not enough. Affected version: 2.8.1...2.8.x, 2.9.x ### Modifications - Skip add the DecimalConversion in `extractAvroSchema()`
codelipenghui
pushed a commit
that referenced
this pull request
Jan 30, 2022
### Motivation I can use Avro schema with BigDecimal in Pulsar 2.8.0, but this doesn't work on Pulsar 2.8.1, so I check this codebase and PR, found #10428 breaks this. The following is error log: ``` org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154) ~[spring-beans-5.3.4.jar:5.3.4] ... 34 common frames omitted Caused by: java.lang.UnsupportedOperationException: No recommended schema for decimal (scale is required) at org.apache.pulsar.shade.org.apache.avro.Conversions$DecimalConversion.getRecommendedSchema(Conversions.java:73) ~[pulsar-client-2.8.1.jar:2.8.1] at org.apache.pulsar.shade.org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:696) ~[pulsar-client-2.8.1.jar:2.8.1] at org.apache.pulsar.shade.org.apache.avro.reflect.ReflectData.createFieldSchema(ReflectData.java:873) ~[pulsar-client-2.8.1.jar:2.8.1] at org.apache.pulsar.shade.org.apache.avro.reflect.ReflectData$AllowNull.createFieldSchema(ReflectData.java:92) ~[pulsar-client-2.8.1.jar:2.8.1] at org.apache.pulsar.shade.org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:736) ~[pulsar-client-2.8.1.jar:2.8.1] at org.apache.pulsar.shade.org.apache.avro.specific.SpecificData$3.computeValue(SpecificData.java:328) ~[pulsar-client-2.8.1.jar:2.8.1] at org.apache.pulsar.shade.org.apache.avro.specific.SpecificData$3.computeValue(SpecificData.java:325) ~[pulsar-client-2.8.1.jar:2.8.1] at java.base/java.lang.ClassValue.getFromHashMap(ClassValue.java:228) ~[na:na] at java.base/java.lang.ClassValue.getFromBackup(ClassValue.java:210) ~[na:na] at java.base/java.lang.ClassValue.get(ClassValue.java:116) ~[na:na] at org.apache.pulsar.shade.org.apache.avro.specific.SpecificData.getSchema(SpecificData.java:339) ~[pulsar-client-2.8.1.jar:2.8.1] ... 52 common frames omitted ``` I think that Avro cannot work on using the ReflectData with Conversions.DecimalConversion to parse the BigDecimal field without AvroSchema when getting schema, but the Avro ReflectDatumWriter and ReflectDatumReader are working, it seems that Avro support for BigDecimal is not enough. Affected version: 2.8.1...2.8.x, 2.9.x ### Modifications - Skip add the DecimalConversion in `extractAvroSchema()` (cherry picked from commit 2ca8e8a)
zymap
pushed a commit
that referenced
this pull request
Jan 30, 2022
### Motivation I can use Avro schema with BigDecimal in Pulsar 2.8.0, but this doesn't work on Pulsar 2.8.1, so I check this codebase and PR, found #10428 breaks this. The following is error log: ``` org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154) ~[spring-beans-5.3.4.jar:5.3.4] ... 34 common frames omitted Caused by: java.lang.UnsupportedOperationException: No recommended schema for decimal (scale is required) at org.apache.pulsar.shade.org.apache.avro.Conversions$DecimalConversion.getRecommendedSchema(Conversions.java:73) ~[pulsar-client-2.8.1.jar:2.8.1] at org.apache.pulsar.shade.org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:696) ~[pulsar-client-2.8.1.jar:2.8.1] at org.apache.pulsar.shade.org.apache.avro.reflect.ReflectData.createFieldSchema(ReflectData.java:873) ~[pulsar-client-2.8.1.jar:2.8.1] at org.apache.pulsar.shade.org.apache.avro.reflect.ReflectData$AllowNull.createFieldSchema(ReflectData.java:92) ~[pulsar-client-2.8.1.jar:2.8.1] at org.apache.pulsar.shade.org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:736) ~[pulsar-client-2.8.1.jar:2.8.1] at org.apache.pulsar.shade.org.apache.avro.specific.SpecificData$3.computeValue(SpecificData.java:328) ~[pulsar-client-2.8.1.jar:2.8.1] at org.apache.pulsar.shade.org.apache.avro.specific.SpecificData$3.computeValue(SpecificData.java:325) ~[pulsar-client-2.8.1.jar:2.8.1] at java.base/java.lang.ClassValue.getFromHashMap(ClassValue.java:228) ~[na:na] at java.base/java.lang.ClassValue.getFromBackup(ClassValue.java:210) ~[na:na] at java.base/java.lang.ClassValue.get(ClassValue.java:116) ~[na:na] at org.apache.pulsar.shade.org.apache.avro.specific.SpecificData.getSchema(SpecificData.java:339) ~[pulsar-client-2.8.1.jar:2.8.1] ... 52 common frames omitted ``` I think that Avro cannot work on using the ReflectData with Conversions.DecimalConversion to parse the BigDecimal field without AvroSchema when getting schema, but the Avro ReflectDatumWriter and ReflectDatumReader are working, it seems that Avro support for BigDecimal is not enough. Affected version: 2.8.1...2.8.x, 2.9.x ### Modifications - Skip add the DecimalConversion in `extractAvroSchema()` (cherry picked from commit 2ca8e8a)
bharanic-dev
pushed a commit
to bharanic-dev/pulsar
that referenced
this pull request
Mar 18, 2022
Nicklee007
pushed a commit
to Nicklee007/pulsar
that referenced
this pull request
Apr 20, 2022
### Motivation I can use Avro schema with BigDecimal in Pulsar 2.8.0, but this doesn't work on Pulsar 2.8.1, so I check this codebase and PR, found apache#10428 breaks this. The following is error log: ``` org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154) ~[spring-beans-5.3.4.jar:5.3.4] ... 34 common frames omitted Caused by: java.lang.UnsupportedOperationException: No recommended schema for decimal (scale is required) at org.apache.pulsar.shade.org.apache.avro.Conversions$DecimalConversion.getRecommendedSchema(Conversions.java:73) ~[pulsar-client-2.8.1.jar:2.8.1] at org.apache.pulsar.shade.org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:696) ~[pulsar-client-2.8.1.jar:2.8.1] at org.apache.pulsar.shade.org.apache.avro.reflect.ReflectData.createFieldSchema(ReflectData.java:873) ~[pulsar-client-2.8.1.jar:2.8.1] at org.apache.pulsar.shade.org.apache.avro.reflect.ReflectData$AllowNull.createFieldSchema(ReflectData.java:92) ~[pulsar-client-2.8.1.jar:2.8.1] at org.apache.pulsar.shade.org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:736) ~[pulsar-client-2.8.1.jar:2.8.1] at org.apache.pulsar.shade.org.apache.avro.specific.SpecificData$3.computeValue(SpecificData.java:328) ~[pulsar-client-2.8.1.jar:2.8.1] at org.apache.pulsar.shade.org.apache.avro.specific.SpecificData$3.computeValue(SpecificData.java:325) ~[pulsar-client-2.8.1.jar:2.8.1] at java.base/java.lang.ClassValue.getFromHashMap(ClassValue.java:228) ~[na:na] at java.base/java.lang.ClassValue.getFromBackup(ClassValue.java:210) ~[na:na] at java.base/java.lang.ClassValue.get(ClassValue.java:116) ~[na:na] at org.apache.pulsar.shade.org.apache.avro.specific.SpecificData.getSchema(SpecificData.java:339) ~[pulsar-client-2.8.1.jar:2.8.1] ... 52 common frames omitted ``` I think that Avro cannot work on using the ReflectData with Conversions.DecimalConversion to parse the BigDecimal field without AvroSchema when getting schema, but the Avro ReflectDatumWriter and ReflectDatumReader are working, it seems that Avro support for BigDecimal is not enough. Affected version: 2.8.1...2.8.x, 2.9.x ### Modifications - Skip add the DecimalConversion in `extractAvroSchema()`
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #10427
Motivation
As AVRO support the UUID type (see AVRO spec), add UUID support for the pulsar AvroSchema.
It also adds the AVRO Decimal logical type (see AVRO spec).
Modifications
Add the logical type conversion to the ReflectData instance when parsing the provided POJO.
Verifying this change
See the provided unit test in pulsar-client/src/test/java/org/apache/pulsar/client/impl/schema/AvroSchemaTest.java