Skip to content

[FLINK-39136][filesystems] Bump google-cloud-storage to 2.68.0 in flink-gs-fs-hadoop#28286

Merged
MartijnVisser merged 2 commits into
apache:masterfrom
MartijnVisser:FLINK-39136-gcs-upgrade
Jun 3, 2026
Merged

[FLINK-39136][filesystems] Bump google-cloud-storage to 2.68.0 in flink-gs-fs-hadoop#28286
MartijnVisser merged 2 commits into
apache:masterfrom
MartijnVisser:FLINK-39136-gcs-upgrade

Conversation

@MartijnVisser
Copy link
Copy Markdown
Contributor

What is the purpose of the change

flink-gs-fs-hadoop bundled google-cloud-storage 2.29.1, which throws a
NullPointerException instead of retrying certain GCS 503 Service Unavailable
errors during resumable uploads. This breaks checkpointing for jobs that write to
gs:// through a RecoverableWriter (e.g. the FileSink). The upstream fix is in
googleapis/java-storage#2987,
which is included in newer releases of the library.

This PR takes over the stale #27679 (thanks @jonchase) and completes it by also
regenerating the bundled-dependency NOTICE, which is required because this module
shades all of its dependencies into the plugin jar.

Brief change log

  • Bump google-cloud-storage 2.29.1 -> 2.68.0 (latest stable) and the matching
    grpc artifacts 1.59.1 -> 1.81.0 in flink-gs-fs-hadoop/pom.xml.
  • Regenerate META-INF/NOTICE to match the bundled dependency set produced by the
    shade plugin (verified to match exactly via tools/ci/license_check.sh).
  • Add the bundled license file for the newly bundled org.codehaus.woodstox:stax2-api.
  • Update the google-cloud-storage version link in the GCS filesystem docs
    (English + Chinese).

Verifying this change

This change is covered by the existing flink-gs-fs-hadoop tests (236 tests pass,
including GSRecoverableWriterTest, the committer/serializer tests, and the
LocalStorageHelper-based GSBlobStorageImplTest, confirming the pinned
google-cloud-nio test dependency stays compatible).

In addition, the license/NOTICE was validated locally with
tools/ci/license_check.sh (no severe issues; the NOTICE matches the 98 bundled
dependencies exactly) and the dependency-convergence and ban-unsafe-jackson
enforcers pass under -Pcheck-convergence.

Finally, the upgraded SDK was manually verified against a real GCS bucket: a
RecoverableWriter wrote a multi-chunk object, took a checkpoint via persist(),
recovered from that checkpoint via recover(), committed, and the committed object
was read back and asserted byte-for-byte (exactly-once), with no NPE/503 failure.

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): yes (upgrades
    google-cloud-storage and grpc, with corresponding NOTICE/license updates)
  • The public API, i.e., is any changed class annotated with @Public(Evolving): no
  • The serializers: no
  • The runtime per-record code paths (performance sensitive): no
  • Anything that affects deployment or recovery: JobManager (and its components),
    Checkpointing, Kubernetes/Yarn, ZooKeeper: yes (improves reliability of GCS
    RecoverableWriter checkpoint/recovery by fixing 503 retry handling)
  • The S3 file system connector: no

Documentation

  • Does this pull request introduce a new feature? no
  • If yes, how is the feature documented? not applicable

Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Generated-by: Claude Code (Opus 4.8)

@flinkbot
Copy link
Copy Markdown
Collaborator

flinkbot commented Jun 1, 2026

CI report:

Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

Copy link
Copy Markdown
Contributor

@spuru9 spuru9 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

- org.apache.httpcomponents:httpclient:4.5.13
- org.apache.httpcomponents:httpcore:4.4.14
- org.conscrypt:conscrypt-openjdk-uber:2.5.2
- org.jspecify:jspecify:1.0.0
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we need it?
AFAIK it is only annotations for compile time checks

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Excluded it from the shaded jar and dropped it from the NOTICE, consistent with the other annotation-only dependencies already excluded in this module (checker-qual, error_prone_annotations, j2objc-annotations).

One nuance: jspecify's annotations are actually @Retention(RUNTIME) rather than CLASS/SOURCE, but they're static-analysis nullness markers that nothing in the gcs-connector / google-cloud-storage stack reads reflectively at runtime, so excluding them is safe. Verified the module still builds, the unit tests pass, and the newly added real-GCS RecoverableWriter/FileSystem ITCases pass against a live bucket.

@github-actions github-actions Bot added the community-reviewed PR has been reviewed by the community. label Jun 1, 2026
…nk-gs-fs-hadoop

The GCS file system bundled google-cloud-storage 2.29.1, which throws a
NullPointerException instead of retrying certain GCS 503 Service Unavailable
errors during resumable uploads, breaking checkpointing for jobs writing to
gs:// via a RecoverableWriter. The upstream fix is in googleapis/java-storage#2987.

Bump google-cloud-storage 2.29.1 -> 2.68.0 and the matching grpc artifacts
1.59.1 -> 1.81.0, regenerate the bundled-dependency NOTICE accordingly, add the
bundled license file for the newly bundled stax2-api, and update the version
links in the GCS filesystem documentation.

The newly pulled-in jspecify dependency provides only static-analysis nullness
annotations that are not needed at runtime, so it is excluded from the shaded
jar like the other annotation-only dependencies (checker-qual,
error_prone_annotations, j2objc-annotations).

Generated-by: Claude Code (Opus 4.8)
@MartijnVisser MartijnVisser force-pushed the FLINK-39136-gcs-upgrade branch from 0d2fd30 to bb16e08 Compare June 1, 2026 18:45
…em against a real bucket

Add integration tests that run against a real GCS bucket, mirroring the existing
S3 filesystem integration tests. They are skipped unless a bucket is configured
via the IT_CASE_GCS_BUCKET environment variable; authentication uses Application
Default Credentials (GOOGLE_APPLICATION_CREDENTIALS).

  - GSTestCredentials gates the tests on IT_CASE_GCS_BUCKET.
  - GSFileSystemBehaviorITCase runs the shared FileSystemBehaviorTestSuite.
  - GSRecoverableWriterITCase exercises the write / persist (checkpoint) /
    recover / commit flow that backs exactly-once FileSink checkpointing on GCS.

Generated-by: Claude Code (Opus 4.8)
@MartijnVisser MartijnVisser force-pushed the FLINK-39136-gcs-upgrade branch from bb16e08 to d36b8b6 Compare June 2, 2026 09:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

community-reviewed PR has been reviewed by the community.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants