Skip to content

Unblock usage of gcs-connector 3.x#5730

Merged
clairemcginty merged 2 commits intomainfrom
unblock-gcs-connector-3
Jul 22, 2025
Merged

Unblock usage of gcs-connector 3.x#5730
clairemcginty merged 2 commits intomainfrom
unblock-gcs-connector-3

Conversation

@clairemcginty
Copy link
Copy Markdown
Contributor

gcs-connector 3 has some interesting new features, namely vectored read support for Parquet and a new FAdvise mode. However, we're blocked from upgrading to it on the 0.14.x branch, since it's compiled against Java 11. Users on Java 11+ were blocked from overriding their project versions as we weren't properly catching a linkage error:

Exception in thread "main" java.lang.NoSuchMethodError: 'com.google.cloud.hadoop.gcsio.GoogleCloudStorageReadOptions$Builder com.google.cloud.hadoop.gcsio.GoogleCloudStorageReadOptions$Builder.setFastFailOnNotFound(boolean)'
	at com.spotify.scio.ScioContext$$anonfun$9$$anonfun$apply$8.apply(ScioContext.scala:475)
	at com.spotify.scio.ScioContext$$anonfun$9$$anonfun$apply$8.apply(ScioContext.scala:475)
	at scala.Option.fold(Option.scala:263)
	at com.spotify.scio.ScioContext$$anonfun$9.apply(ScioContext.scala:475)
	at com.spotify.scio.ScioContext$$anonfun$9.apply(ScioContext.scala:472)
	at scala.util.ChainingOps$.pipe$extension(ChainingOps.scala:64)
	at com.spotify.scio.ScioContext.<init>(ScioContext.scala:472)
	at com.spotify.scio.ContextAndArgs$$anonfun$withParser$1.apply(ScioContext.scala:219)
	at com.spotify.scio.ContextAndArgs$$anonfun$withParser$1.apply(ScioContext.scala:209)
	at com.spotify.scio.ContextAndArgs$.apply(ScioContext.scala:224)
	at com.spotify.data.example.ParquetExample$.pipeline(ParquetExample.scala:26)
	at com.spotify.data.example.ParquetExample$.main(ParquetExample.scala:45)
	at com.spotify.data.example.ParquetExample.main(ParquetExample.scala)
[error] nonzero exit code returned from runner: 1

@clairemcginty clairemcginty requested review from charles-tan and jto July 21, 2025 20:09
@codecov
Copy link
Copy Markdown

codecov bot commented Jul 21, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 61.47%. Comparing base (05379a9) to head (22d65a2).
Report is 1 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #5730      +/-   ##
==========================================
+ Coverage   61.46%   61.47%   +0.01%     
==========================================
  Files         314      314              
  Lines       11326    11326              
  Branches      792      792              
==========================================
+ Hits         6961     6963       +2     
+ Misses       4365     4363       -2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@clairemcginty clairemcginty merged commit 17fcd8c into main Jul 22, 2025
12 checks passed
@clairemcginty clairemcginty deleted the unblock-gcs-connector-3 branch July 22, 2025 19:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants