[Bug]: Dependency Issue with Apache Beam 2.51.0 Upgrade

[anilmor23]

What happened?

After upgrading Apache Beam from version 2.44.0 to 2.51.0, we are encountering a dependency issue related to the Google Cloud BigQuery library. Specifically, we are facing the following error:

org.apache.beam.sdk.util.UserCodeException: java.lang.NoClassDefFoundError: Could not initialize class com.google.cloud.bigquery.storage.v1.stub.GrpcBigQueryWriteStub

We attempted to resolve this issue by trying different versions of the google-cloud-bigquery dependency, such as 2.29.0, but none of them worked. Even after downgrading to Apache Beam 2.50.0, the issue persists. The only resolution is to revert to Apache Beam 2.44.0, which functions correctly. But Dataflow 2.44.0 has a dataloss issue.

This bug report aims to address the compatibility issue between Apache Beam 2.51.0 and the Google Cloud BigQuery library, seeking a solution or workaround for this problem to enable successful use of the latest Apache Beam version.

Issue Priority

Priority: 0 (outage / urgent vulnerability)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[Abacn]

Hi, are you using a custom container? Also, if you are using maven or gradle, could use standard maven/gradle tasks to check the dependency tree, to see why old version of bigquery-storage client was picked up.

This is almost likely a configuration issue, set to P2.