Skip to content

Support Avro 1.11 java-class annotations in scio-parquet#5821

Merged
clairemcginty merged 5 commits intomainfrom
support-java-class-annotation-parquet
Dec 4, 2025
Merged

Support Avro 1.11 java-class annotations in scio-parquet#5821
clairemcginty merged 5 commits intomainfrom
support-java-class-annotation-parquet

Conversation

@clairemcginty
Copy link
Copy Markdown
Contributor

parquet-avro requires a default configuration option to be set to read and write Avro classes using java-class property: https://github.com/clairemcginty/parquet-mr/blob/master/parquet-avro/README.md#configuration-for-reading

Comment on lines +33 to +35
<property>
<name>parquet.avro.compatible</name>
<value>false</value>
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This option basically toggles Parquet to use its "new" Avro record materializer, AvroRecordMaterializer, instead of the legacy one, AvroIndexedRecordMaterializer. From what I can tell, AvroRecordMaterializer has a superset of AvroIndexedRecordConverter's features, so it should be safe to use.

In any case, only AvroRecordMaterializer actually parses the parquet.avro.serializable.classes option to pass in for the Avro reflection bit, so we need it to make the java-class functionality work.

@codecov
Copy link
Copy Markdown

codecov bot commented Dec 2, 2025

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 61.48%. Comparing base (041ae06) to head (a2ae6ca).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #5821      +/-   ##
==========================================
+ Coverage   61.46%   61.48%   +0.02%     
==========================================
  Files         314      314              
  Lines       11431    11437       +6     
  Branches      828      830       +2     
==========================================
+ Hits         7026     7032       +6     
  Misses       4405     4405              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@clairemcginty clairemcginty requested a review from kellen December 2, 2025 17:21
Copy link
Copy Markdown
Contributor

@kellen kellen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yolo

@clairemcginty clairemcginty merged commit b566764 into main Dec 4, 2025
12 checks passed
@clairemcginty clairemcginty deleted the support-java-class-annotation-parquet branch December 4, 2025 20:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants