Support Avro 1.11 java-class annotations in scio-parquet#5821
Support Avro 1.11 java-class annotations in scio-parquet#5821clairemcginty merged 5 commits intomainfrom
Conversation
| <property> | ||
| <name>parquet.avro.compatible</name> | ||
| <value>false</value> |
There was a problem hiding this comment.
This option basically toggles Parquet to use its "new" Avro record materializer, AvroRecordMaterializer, instead of the legacy one, AvroIndexedRecordMaterializer. From what I can tell, AvroRecordMaterializer has a superset of AvroIndexedRecordConverter's features, so it should be safe to use.
In any case, only AvroRecordMaterializer actually parses the parquet.avro.serializable.classes option to pass in for the Avro reflection bit, so we need it to make the java-class functionality work.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #5821 +/- ##
==========================================
+ Coverage 61.46% 61.48% +0.02%
==========================================
Files 314 314
Lines 11431 11437 +6
Branches 828 830 +2
==========================================
+ Hits 7026 7032 +6
Misses 4405 4405 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
parquet-avro requires a default configuration option to be set to read and write Avro classes using
java-classproperty: https://github.com/clairemcginty/parquet-mr/blob/master/parquet-avro/README.md#configuration-for-reading