v0.15.9

(Mostly) bugfix release addressing the same bug as in 0.15.8 because scio delivers 2 core-site xml files which means the JVM may return either of them when hadoop constructs a new Configuration(). If the scio-smb core-site resource was loaded, vectored reads were (again) enabled by default causing crashes when reading from GCS on dataflow.

🚀 Enhancements

• Add staging.jars.allowedenvdirs setting by @kellen in #5958

🐛 Bug Fixes

• Sync scio-smb core-site.xml and add parity test by @kellen in #5957

🌱 Dependency Updates

• Bump Magnolify to 0.9.5 by @clairemcginty in #5956

🏗️ Build Improvements

• Fix CI publish race condition in clean workflow by @kellen in #5981

Full Changelog: v0.15.8...v0.15.9

v0.15.8

Bugfix release. v0.15.7 includes a config change in a dependency which enabled vectored reads in GCS, but due to a version incompatibility, on read this results in:

java.lang.NoSuchMethodError: 'void org.apache.hadoop.fs.VectoredReadUtils.validateRangeRequest(org.apache.hadoop.fs.FileRange)'

The config is toggled back to default to false.

🐛 Bug Fixes

• Fix GCS/hadoop read options by @kellen in #5949
• Disable parquet vectored reads to fix gcs-connector incompatibility by @kellen in #5954
• Fix FixBigtableIO scalafix rule mangling named arguments by @kellen in #5948

🏗️ Build Improvements

• Add integration test that should fail by @kellen in #5950
• Fix auth in IT? by @kellen in #5953

Full Changelog: v0.15.7...v0.15.8

v0.15.7

Scio 0.15.7 includes Beam 2.74.0

🚀 Enhancements

• Upgrade to Beam 2.74.0, drop bigdataoss v2, rework checkBeamDependencies by @kellen in #5937
• Aggregate scio-managed project in root so it publishes by @clairemcginty in #5934
• Add additional bigtable scalafix by @kellen in #5938
• (iceberg) Infer keep config from Magnolify RowType schema by @clairemcginty in #5942

🐛 Bug Fixes

• Fix multi-path URI escaping in GcsConnectorUtil by @charles-tan in #5923
• Add timestampSkew to BaseAsyncDoFn.java by @scwhittle in #5933
• fix javafmt issue by @clairemcginty in #5935
• (iceberg) Fix scala-to-java List conversion for managed config by @clairemcginty in #5941
• Don't break avro 1.8 by @kellen in #5943

🏗️ Build Improvements

• Fix repl merge strategy by @kellen in #5944
• Fix site mdoc by @kellen in #5945

🌱 Dependency Updates

• Update elasticsearch-java to 8.19.14 by @clairemcginty in #5924
• Update sbt-java-formatter to 0.11.0 by @clairemcginty in #5925
• Update sbt, sbt-dependency-tree, ... to 1.12.9 by @clairemcginty in #5927
• Update jedis to 7.4.1 by @clairemcginty in #5928

Full Changelog: v0.15.6...v0.15.7

v0.15.6

🐛 Bug Fixes

• Always deep-copy Parquet configs by @clairemcginty in #5920

📗 Documentation

• Add deprecation header by @kellen in #5882

Full Changelog: v0.15.5...v0.15.6

v0.15.5

Scio 0.15.5 includes Beam 2.73

🚀 Enhancements

• Make Voyager index and names filenames configurable by @dylanrb123 in #5914

🌱 Dependency Updates

• Upgrade to Beam 2.72 by @cjordn in #5905
• Update sbt-avro, sbt-avro-compiler-bridge to 4.0.2 by @clairemcginty in #5889
• Update mysql-connector-j to 9.6.0 by @clairemcginty in #5893
• Upgrade SBT to 1.12.7 by @clairemcginty in #5906
• Update sbt-typelevel to 0.8.5 by @clairemcginty in #5915
• Update jedis to 7.2.1 by @clairemcginty in #5917
• Update sbt-scoverage to 2.4.4 by @clairemcginty in #5913
• Update scalafmt-core to 3.10.7 by @clairemcginty in #5911
• Update scalac-options to 0.1.9 by @clairemcginty in #5916
• Update munit to 1.2.4 by @clairemcginty in #5910
• Update scalactic to 3.2.20 by @clairemcginty in #5909
• Update sbt, sbt-dependency-tree, ... to 1.12.8 by @clairemcginty in #5908
• Update scalatest to 3.2.20 by @clairemcginty in #5912
• Bump sparkey version by @spkrka in #5902

Full Changelog: v0.15.4...v0.15.5

v0.15.4

🚀 Enhancements

• Support Java 25 by @clairemcginty in #5883

🏗️ Build Improvements

• Add test to ParquetTestUtilsTest to check list projections and CaseMappers by @clairemcginty in #5900

🌱 Dependency Updates

• Update elasticsearch-java to 8.19.13 by @clairemcginty in #5888
• Update sbt-bom to 1.0.33 by @clairemcginty in #5891
• Update cloud-sql-connector-jdbc-sqlserver, ... to 1.28.2 by @clairemcginty in #5890
• Update neo4j-java-driver to 4.4.22 by @clairemcginty in #5898
• Update parquet-avro, parquet-column, ... to 1.17.0 by @clairemcginty in #5897
• Update mssql-jdbc to 13.4.0.jre11 by @clairemcginty in #5892
• Update metrics-core to 4.2.38 by @clairemcginty in #5895
• Update iceberg-common, iceberg-core, ... to 1.10.1 by @clairemcginty in #5896
• Upgrade Sparkey to 3.5.1 by @clairemcginty in #5899

Full Changelog: v0.15.3...v0.15.4

v0.15.3

Scio 0.15.3 includes Beam 2.71

🐛 Bug Fixes

• (fix #5874) Fix missing field descriptions for typed BigQuery writes by @clairemcginty in #5875
• Rename BufferedExternalSorter to avoid class load conflicts by @shnapz in #5877
• fix dropped AvroType @doc annotations by @clairemcginty in #5879
• Fix record-level Avro doc by @clairemcginty in #5880

🌱 Dependency Updates

• Upgrade to Beam 2.71 by @clairemcginty in #5873
• Use slf4j v2 by @kellen in #5872
• Bump Magnolify to 0.9.3 by @clairemcginty in #5878

Full Changelog: v0.15.2...v0.15.3

v0.15.2

Scio 0.15.2 includes Beam 2.70

🚀 Enhancements

• Enable Beam lineage for JdbcIO by @shnapz in #5849
• Make SMB classes extensible for external libraries by @spkrka in #5869

🐛 Bug Fixes

• Fix SmbIO tap to use FileOperations coder by @spkrka in #5870

🌱 Dependency Updates

• Upgrade to Beam 2.70 by @clairemcginty in #5867
• Update jedis to 7.2.0 by @scala-steward in #5863
• Update sbt-typelevel to 0.8.4 by @scala-steward in #5862
• Update scalafmt-core to 3.10.3 by @scala-steward in #5860
• Update sbt-mdoc to 2.8.2 by @scala-steward in #5859
• Update scala-compiler, scala-library, ... to 2.13.18 by @scala-steward in #5857
• Update scalacheck to 1.19.0 by @scala-steward in #5858
• Update caffeine to 3.2.3 by @scala-steward in #5853
• Update elasticsearch-java to 8.19.9 by @scala-steward in #5851
• Update tensorflow-core-native to 1.1.0 by @scala-steward in #5861
• Update beam-examples-java, ... to 2.69.0 by @clairemcginty in #5806
• Update sbt, sbt-dependency-tree, ... to 1.12.0 by @scala-steward in #5865
• Update sbt-unidoc to 0.6.1 by @scala-steward in #5864
• Update scala-compiler, scala-library, ... to 2.12.21 by @scala-steward in #5856
• Bump checkout/download-artifact/iceberg/testcontainers by @kellen in #5871
• Prepare for 0.15.2 release by @clairemcginty in #5868

Full Changelog: v0.15.1...v0.15.2

v0.15.1

Bugfix release

• Un-deprecate macro annotations to avoid magnolify serialization issues, deprecate typed avro and BQ methods by @kellen in #5846
• Check if org.apache.avro.SERIALIZABLE_CLASSES in scio-smb Parquet writes by @clairemcginty in #5847
• Null check on org.apache.avro.SERIALIZABLE_CLASSES prop by @clairemcginty in #5829

🚀 Enhancements

• scalafix for 0.15 by @kellen in #5822

📗 Documentation

• Add migration guide, deprecations for 0.15 by @kellen in #5845

🌱 Dependency Updates

• Update cloud-sql-connector-jdbc-sqlserver, ... to 1.27.1 by @clairemcginty in #5833
• Update sbt-bom to 1.0.31 by @clairemcginty in #5834
• Update pprint to 0.9.6 by @clairemcginty in #5835
• Update sparkey to 3.5.0 by @clairemcginty in #5836
• Update scalacheck to 1.18.1 by @clairemcginty in #5840
• Update sbt-scoverage to 2.4.3 by @clairemcginty in #5843
• Update sbt-scalafix to 0.14.5 by @clairemcginty in #5830
• Upgrade Sparkey to 3.5.0 by @spkrka in #5827

Full Changelog: v0.15.0...v0.15.1

v0.15.0

Scio 0.15 adds support for Iceberg, beam's managed transforms, and Avro 1.11's java-class annotations.

🚀 Enhancements

• Magnolify API by @kellen in #5286
• (fix #5067) Drop Java 8 Targeting by @clairemcginty in #5203
• Managed + Iceberg IO by @kellen in #5494
• Support writing extra metadata in scio-parquet by @clairemcginty in #5411
• Update IcebergIO read/write parameters by @kellen in #5823
• Add integration test for Magnolify TableRowType compatibility by @clairemcginty in #5818
• Support Avro 1.11 java-class annotations in scio-parquet by @clairemcginty in #5821
• Add ParquetMetadataDoFn by @kellen in #5820
• v0.15.0 by @kellen in #5609
• Remove stale comments, make some previously-deprecated methods private by @kellen in #5584

🏗️ Build Improvements

• Vendor Socco by @kellen in #5811
• Exclude dependabot from updating ci.yml by @kellen in #5813
• Move CI env definition by @kellen in #5826
• Remove developers list by @kellen in #5548
• Fix BT integration tests by @kellen in #5606
• Fix BT integration tests mark ii by @kellen in #5608
• Add headers, remove unused by @kellen in #5619

🐛 Bug Fixes

• Fix Iceberg integration test, defaults by @kellen in #5825
• Fix seed setting in samplers by @kellen in #5816
• Add check for java 17 SCollectionMatcher (fix #5483) by @kellen in #5605

🌱 Dependency Updates

• Upgrade to Avro 1.11.5 by @clairemcginty in #5799
• Update sbt to 1.11.7, sbt-mdoc to 2.8.0, sbt-java-formatter to 0.10.0… by @kellen in #5815
• Bump to magnolify 0.9.2 by @kellen in #5824
• Update scala-compiler, scala-library, ... to 2.13.17 by @clairemcginty in #5786
• Bump coursier/cache-action from 6 to 7 by @dependabot[bot] in #5797
• Update sbt-typelevel to 0.8.2 by @clairemcginty in #5808
• Update sbt-jmh to 0.4.8 by @clairemcginty in #5809
• Update mysql-connector-j to 9.5.0 by @clairemcginty in #5804
• Update cloud-sql-connector-jdbc-sqlserver, ... to 1.27.0 by @clairemcginty in #5802
• Update jedis to 7.0.0 by @clairemcginty in #5810
• Update sbt-scoverage to 2.4.1 by @clairemcginty in #5807
• Update sbt-bom to 1.0.30 by @clairemcginty in #5803
• Update testcontainers-scala-elasticsearch, ... to 0.43.6 by @clairemcginty in #5801
• Update elasticsearch-java to 8.19.7 by @clairemcginty in #5800
• Update magnolify to 0.8.0, tensorflow to 1.0.0, caffeine to 3.2.0 by @kellen in #5604
• magnolify 0.9 by @kellen in #5812

Full Changelog: v0.14.20...v0.15.0