[BEAM-6480] Adds AvroIO sink for generic records.#9005
[BEAM-6480] Adds AvroIO sink for generic records.#9005aromanenko-dev merged 6 commits intoapache:masterfrom
Conversation
|
R: @iemejia Can you take a look? |
sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroIO.java
Outdated
Show resolved
Hide resolved
iemejia
left a comment
iemejia
left a comment
There was a problem hiding this comment.
Looks pretty ok, please apply the suggestions on naming and minor fixes and it will be ready to go.
sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroIO.java
Outdated
Show resolved
Hide resolved
sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroIO.java
Outdated
Show resolved
Hide resolved
| private enum WriteMethod { | ||
| AVROIO_WRITE, | ||
| AVROIO_SINK | ||
| AVROIO_SINK, |
There was a problem hiding this comment.
Ooof -- this is used in different places to distinguish between write and sink techniques (with and without Schema).
I think I'll add one more to distinguish between the three types of sinks: AVROIO_SINK_CLASS with a specific/generated class, AVROIO_SINK_WITH_SCHEMA and AVROIO_SINK_WITH_FORMATTER. What do you think?
sdks/java/core/src/test/java/org/apache/beam/sdk/io/AvroIOTest.java
Outdated
Show resolved
Hide resolved
sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroIO.java
Outdated
Show resolved
Hide resolved
Modify unit test to show that the sink can work directly with a PCollection<GenericRecord> without casting.
|
Oops -- there's a I can either rewrite the test to use a single schema and different criteria to select dynamic destinations (which exercises the code equally well) or write a custom Coder to correctly serialize the different GenericRecords in a single collection (which is a good example of how to use this technique). I'm taking a look. |
|
Run Java PreCommit |
aromanenko-dev
left a comment
aromanenko-dev
left a comment
There was a problem hiding this comment.
Thanks! It almost LGTM, I just left several comments.
sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroIO.java
Outdated
Show resolved
Hide resolved
3 participants
Add a sink to AvroIO that takes generic data (usually GenericRecords). The current method sinkViaGenericRecords() has been deprecated.
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
R: @username).[BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replaceBEAM-XXXwith the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.Post-Commit Tests Status (on master branch)
Pre-Commit Tests Status (on master branch)
See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.