update confluent dependency version to 7.6.0 #30638

tilgalas · 2024-03-14T16:32:56Z

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

github-actions · 2024-03-14T18:06:15Z

Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment assign set of reviewers

Abacn · 2024-03-14T18:57:16Z

There are test failures related to avro code generation dependency change:

java.lang.NoClassDefFoundError: Could not initialize class org.apache.beam.sdk.io.gcp.bigquery.AvroGenericRecordToStorageApiProtoTest
...

and

java.lang.AssertionError: null definition should throw an exception: unexpected exception type thrown; expected:<java.lang.NullPointerException> but was:<org.apache.avro.SchemaParseException>
	at org.junit.Assert.assertThrows(Assert.java:1020)
	at org.apache.beam.sdk.io.gcp.pubsub.PubsubClientTest.fromPubsubSchema(PubsubClientTest.java:206)
Caused by: org.apache.avro.SchemaParseException: Cannot parse <null> schema
	at org.apache.avro.Schema.parse(Schema.java:1692)

see https://github.com/apache/beam/runs/22672196056

buildSrc/build.gradle.kts

sdks/java/extensions/avro/build.gradle

Abacn · 2024-03-14T19:01:07Z

Thanks for the contribution! It involves avro dependency and things get tricky. As avro dependency in Beam is known to be hard to change.

Is it possible to avoid avro dependency changes and still update confluent dependency?

tilgalas · 2024-03-14T19:49:47Z

Thanks for the contribution! It involves avro dependency and things get tricky. As avro dependency in Beam is known to be hard to change.

Is it possible to avoid avro dependency changes and still update confluent dependency?

Thanks for reviewing! I'll try fixing the problems that the tests discovered, if it gets too tricky or risky I imagine we might want to update the confluent version gradually, ie instead of going straight for the 7.6.0 maybe try first with something less radical like 6.x or even 5.4

…izable

Abacn · 2024-03-15T17:34:28Z

...rm/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/AvroGenericRecordToStorageApiProto.java

@@ -262,8 +262,7 @@ private static TableFieldSchema fieldDescriptorFromAvroField(Schema.Field field)
            elementType.getType() != Schema.Type.UNION,
            "Multiple non-null union types are not supported.");
        TableFieldSchema unionFieldSchema =
-            fieldDescriptorFromAvroField(
-                new Schema.Field(field.name(), elementType, field.doc(), field.defaultVal()));


why defaultVal dropped here?

the elementType here will be a schema of the union, but with the null type filtered out - in the "extreme" case of an "optionable" union type of [null, SomeSchema] it won't even be a union anymore, just the bare SomeSchema - if the defaultVal is null, it will not correspond to this derived schema anymore, and Avro will throw an exception when constructing the Field object - I'm not sure why this wasn't a problem before (maybe Avro didn't actually check this?)

Abacn · 2024-03-15T17:42:03Z

buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy

-        avro                                        : "org.apache.avro:avro:1.8.2",
-        avro_tests                                  : "org.apache.avro:avro:1.8.2:tests",
+        avro                                        : "org.apache.avro:avro:1.11.3",
+        avro_tests                                  : "org.apache.avro:avro:1.11.3:tests",


I remember there was issue upgrading avro version and that was why it sticked to 1.8.2 for a long time. Might be good to ask in devlist about this proposal

sdks/java/extensions/avro/build.gradle

...tensions/avro/src/main/java/org/apache/beam/sdk/extensions/avro/schemas/utils/AvroUtils.java

Abacn · 2024-03-15T17:56:22Z

also, it's good to run https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_Avro_Versions.yml in this PR. You can add an empty file named .github/trigger_files/beam_PostCommit_Java_Avro_Versions.json to trigger the test, see README here: https://github.com/apache/beam/tree/master/.github/workflows

tilgalas · 2024-03-15T20:16:44Z

Run Java_GCP_IO_Direct PreCommit

Abacn

Tests now all passed.

wait for another day and will merge if no objection in the devlist mail thread

Abacn · 2024-03-22T16:37:53Z

There is (only) one integration test failing, not too bad:

testBigQueryStorageReadProjectionPushdown (org.apache.beam.sdk.io.gcp.bigquery.BigQueryIOStorageReadIT) failed

java.lang.RuntimeException: org.apache.beam.sdk.util.UserCodeException: java.lang.IllegalArgumentException: Error converting field Field{name=string_field, description=, type=STRING, options={{}}}: Not a valid schema field: string_field
...
Caused by: org.apache.avro.AvroRuntimeException: Not a valid schema field: string_field
	at org.apache.avro.generic.GenericData$Record.get(GenericData.java:282)
	at org.apache.beam.sdk.io.gcp.bigquery.BigQueryUtils.lambda$toBeamRow$5(BigQueryUtils.java:500)

See https://github.com/apache/beam/runs/22988286765

* update confluent dependency version to 7.6.0 * make avroSchema field non-transient as the Schema class is now Serializable * remove now unused org.codehaus.jackson package from GcpApiSurfaceTest * update expected exception class in PubsubClientTest * fix AvroGenericRecordToStorageApiProtoTest * move org.tukaani:xz:1.9 dependency to testImplementation * bring back the tukaani dep to the avroVersions configurations * trigger the beam_PostCommitJava_Avro_Versions workflow

update confluent dependency version to 7.6.0

e95e715

github-actions bot added java build io extensions gcp kafka labels Mar 14, 2024

Abacn reviewed Mar 14, 2024

View reviewed changes

buildSrc/build.gradle.kts Show resolved Hide resolved

Abacn reviewed Mar 14, 2024

View reviewed changes

sdks/java/extensions/avro/build.gradle Outdated Show resolved Hide resolved

tilgalas added 5 commits March 14, 2024 20:30

make avroSchema field non-transient as the Schema class is now Serial…

18ac188

…izable

remove now unused org.codehaus.jackson package from GcpApiSurfaceTest

b51a96f

update expected exception class in PubsubClientTest

4e91c0d

fix AvroGenericRecordToStorageApiProtoTest

824a076

move org.tukaani:xz:1.9 dependency to testImplementation

fd0d50d

Abacn reviewed Mar 15, 2024

View reviewed changes

tilgalas added 2 commits March 15, 2024 19:03

bring back the tukaani dep to the avroVersions configurations

6839a3d

trigger the beam_PostCommitJava_Avro_Versions workflow

11dd82f

tilgalas force-pushed the kafka-dep branch from bf5b5ae to 11dd82f Compare March 15, 2024 19:09

Abacn approved these changes Mar 18, 2024

View reviewed changes

Abacn merged commit 764fcda into apache:master Mar 19, 2024
32 checks passed

Abacn mentioned this pull request Apr 2, 2024

Add back Avro 1.8.2 to test #30833

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

update confluent dependency version to 7.6.0 #30638

update confluent dependency version to 7.6.0 #30638

tilgalas commented Mar 14, 2024 •

edited

github-actions bot commented Mar 14, 2024

Abacn commented Mar 14, 2024

Abacn commented Mar 14, 2024

tilgalas commented Mar 14, 2024

Abacn Mar 15, 2024

tilgalas Mar 15, 2024 •

edited

Abacn Mar 15, 2024

Abacn commented Mar 15, 2024

tilgalas commented Mar 15, 2024

Abacn left a comment

Abacn commented Mar 22, 2024

update confluent dependency version to 7.6.0 #30638

update confluent dependency version to 7.6.0 #30638

Conversation

tilgalas commented Mar 14, 2024 • edited

GitHub Actions Tests Status (on master branch)

github-actions bot commented Mar 14, 2024

Abacn commented Mar 14, 2024

Abacn commented Mar 14, 2024

tilgalas commented Mar 14, 2024

Abacn Mar 15, 2024

Choose a reason for hiding this comment

tilgalas Mar 15, 2024 • edited

Choose a reason for hiding this comment

Abacn Mar 15, 2024

Choose a reason for hiding this comment

Abacn commented Mar 15, 2024

tilgalas commented Mar 15, 2024

Abacn left a comment

Choose a reason for hiding this comment

Abacn commented Mar 22, 2024

tilgalas commented Mar 14, 2024 •

edited

tilgalas Mar 15, 2024 •

edited