Dataflow Templates 2026-09-29-00_RC02

add parquet to lake house blueprint and IT (#4353)

* update lakehouse options

* add parquet to lakehouse yaml pipeline

* update provider syntax

* add projectId for catalog configs

* add comment about gcs file path

Dataflow Templates 2026-09-22-00_RC00

What's Changed

Bug fixes

• test: de-flake AstraDb, reverse-migration, and datastream job-init ITs by @manitgupta in #4260
• Fix PostgreSQL inheritance IT to exercise CDC instead of backfill by @manitgupta in #4261
• Retry transient Cloud SQL connection failures in CloudSqlResourceManager by @manitgupta in #4263
• Fix resource exhausted errors in gcs-spanner-dv tests by @aasthabharill in #4268
• fix: do not emit inline column OPTIONS for postgresql dialect by @sm745052 in #4276

Improvements

• [SpannerToSourceDb] Removing pwd input validation in Reverse replication and Bulk templates by @darshan-sj in #4264

Full Changelog: 2026-09-15-00_RC00...2026-09-22-00_RC00

Dataflow Templates 2026-09-15-00_RC00

What's Changed

Additional New Features

• feat(Spanner): Bulk Migration: add SQL Server source support including connector implementation, dialect adapter, and configuration mappings by @pratickchokhani in #4185
• feat(Spanner): Reverse: implement initial SQL Server source connector and DML generation support for Spanner-to-SourceDB migration by @pratickchokhani in #4234
• feat(Spanner): Live Migration: add SQL Server source support including connector implementation, dialect adapter, and configuration mappings by @pratickchokhani in #4237
• test(Spanner): Bulk Migration: Implemented data type integration test for Sql Server by @pratickchokhani in #4239
• [Data Validations] Change public doc link by @aasthabharill in #4247
• [Data Validation] Add table configuration by @aasthabharill in #4188
• added skills for implementing a source on data validation by @bharadwaj-aditya in #4219

Bug fixes

• fix(spanner): skip locality group option in information schema until full support by @finnzzf in #4216
• test: disable flaky SQLServer bulk-migration ITs by @manitgupta in #4259
• Updates the DeltaLake to Lakehouse template and the test... by @chamikaramj in #4258

Improvements

• Rename validation pipeline by @aasthabharill in #4242
• Update to beam 2.76 by @damccorm in #4203
• fix: address CVE-2026-54399 vulnerability in httpcore5 by @tvalentyn in #4246
• Pin lz4-java to a newer version. by @tvalentyn in #4248
• feat(mongodb-to-mongodb): High-Throughput Backfill Migration (with Experimental CDC) by @michaeltle-goog in #4235

New Contributors

• @finnzzf made their first contribution in #4216

Full Changelog: 2026-09-08-00_RC01...2026-09-15-00_RC00

Dataflow Templates 2026-09-08-00_RC01

What's Changed

Additional New Features

• add auto-close dependabot step for requirements*.txt type files by @derrickaw in #4220
• add delta lake to iceberg blueprint and IT by @derrickaw in #4103
• Add oracle bulk ITs by @dhwanilpatel in #4161
• Add automated PR watchdog for codelab parameters by @aasthabharill in #4206
• refactor: add Jdbc url to shard pojo. This will be used by jdbc connection helper. It will allow the URL to be created by the respective source connector. by @pratickchokhani in #4224

Bug fixes

• Fix Unit Test failure by @aasthabharill in #4226

Improvements

• fix naming determination by @derrickaw in #4210
• Refactor pipeline parameter options to seperate Options.java file by @aasthabharill in #4209
• Remove optional tag from mandatory template params by @aasthabharill in #4225
• improvement: expose wait timeout by @mutianf in #4221

Full Changelog: 2026-09-01-00_RC00...2026-09-08-00_RC01

Dataflow Templates 2026-09-01-00_RC00

What's Changed

Additional New Features

• feat(yaml): Modernize PubSub to BigQuery streaming template with YAML by @ihebimad in #4168
• feat(yaml): Modernize PostgreSQL to BigQuery template with YAML by @ihebimad in #4145
• added oracle support to data validation by @bharadwaj-aditya in #4173
• fixed oracle integration test by @bharadwaj-aditya in #4187
• Terraform module for gcs-spanner-dv by @aasthabharill in #4183

Bug fixes

• fix java text by @derrickaw in #4197

Improvements

• [SpannerToSourceDb] Add support for URL-encoded connection properties and enable MySQL SSL based authentication by @darshan-sj in #4184

Full Changelog: 2026-08-25-00_RC00...2026-09-01-00_RC00

Dataflow Templates 2026-08-25-00_RC00

What's Changed

Additional New Features

• Add Bulk support for ORACLE by @dhwanilpatel in #4133
• Add Reverse support for ORACLE by @dhwanilpatel in #4137
• feat(mongodb-to-mongodb]: Read/Write Performance and Reliability Improvements by @michaeltle-goog in #4079
• [Data Validation] Postgres AllDatatypes E2E test by @aasthabharill in #4138

Bug fixes

• remove dependabot labeling workflow and add step to release notes label workflow to prevent spurious failures by @derrickaw in #4164
• Min CPU count 4 added by @pratickchokhani in #4131
• fix(Spanner): Segregate spanner instances for templates in load test. by @pratickchokhani in #4136

Improvements

• Remove jline static Log; use SLF4J Logger by @kennknowles in #4143
• update upload jar workflow parameters etc by @derrickaw in #4053
• add hadoop aws dependencies by @derrickaw in #4162
• update jar file path from 0.2.0 -> 0.2.2 to include aws hadoop dependencies by @derrickaw in #4167

Package Upgrades

• Bump netty version to 4.2.17.Final by @kennknowles in #4142
• Bump activemq version to 5.19.9 by @kennknowles in #4155
• Exclude jline from transitive dependencies of hadoop by @kennknowles in #4148

New Contributors

• @kennknowles made their first contribution in #4142

Full Changelog: 2026-08-18-00_RC00...2026-08-25-00_RC00

Dataflow Templates 2026-08-18-00_RC00

What's Changed

Additional New Features

• [Data Validation] Duplicate Rows + Generated Spanner col IT by @aasthabharill in #4116
• removed sourcedb to spanner from java pr as it is present in the span… by @bharadwaj-aditya in #4134
• Spanner to spanner transactional handling fixes by @bharadwaj-aditya in #4127
• [Data Validation] AllDatatypes E2E test by @aasthabharill in #4112
• Add migration source skills by @bharadwaj-aditya in #4091

Bug fixes

• Fix Github failures due to BulkMigrationAndValidationMySQLAllDataTypesE2EIT by @aasthabharill in #4144

Improvements

• Refactor spanner-staging-tests by @aasthabharill in #4017

Full Changelog: 2026-08-11-00_RC00...2026-08-18-00_RC00

Dataflow Templates 2026-08-11-00_RC00

What's Changed

Additional New Features

• feat(sourcedb-to-spanner): add mapping support for missing postgresql data types by @jsuhani-2026 in #4075
• docs(sourcedb-to-spanner): update PostgreSQL limitations and supported types by @jsuhani-2026 in #4089
• feat: implement PostgreSQL MONEY data type parsing and integration tests by @jsuhani-2026 in #4104
• Support PostgreSQL INTERVAL, TIMETZ, BYTEA, and BIT types in Live Migration and expand IT coverage by @jsuhani-2026 in #4095
• Add source migration testing skills by @dhwanilpatel in #4113
• feat(spanner-to-sourcedb): Classify NOT_FOUND as retryable to support interleaved table replication by @srozsnyai in #4045
• feat(Spanner): integrate SourceConfigParser to centralize shard configuration loading for SourceDbToSpanner pipelines. by @pratickchokhani in #3854

Bug fixes

• Revert "build(deps): bump cryptography from 47.0.0 to 50.0.0 in /yaml/src/main/python" by @derrickaw in #4114
• fix(datastream-mongodb-to-firestore): handle null-data UPDATE events and improve transient retry classification by @namita-l in #4111

Improvements

• Enable Apache Iceberg REST Metrics Reporting for Lakehouse by @feefs in #4115
• Switch Lakehouse from v1beta to v1 by @feefs in #4118

Package Upgrades

• Bump jackson version to 2.21.5 by @Abacn in #4125

New Contributors

• @feefs made their first contribution in #4115

Full Changelog: 2026-08-04-00_RC00...2026-08-11-00_RC00

Dataflow Templates 2026-08-04-00_RC00

What's Changed

Additional New Features

• feat(sourcedb-to-spanner): Add uniform splitting support for extended… by @jsuhani-2026 in #4028
• [Data Validation] Add Custom Transformation ITs by @aasthabharill in #4062
• Adds Azure dependencies to the YAML template by @chamikaramj in #4088
• [Data Validation] Sharded E2E test by @aasthabharill in #4076
• [Data Validation] Add Primary Key IT by @aasthabharill in #4077
• Add Datastream to Spanner IT for PostgreSQL partitioned tables by @jsuhani-2026 in #4081
• [Data Validation] Max Columns Wide Row IT by @aasthabharill in #4083
• [Data Validation] Add wide row tests for max column and table name lengths by @aasthabharill in #4093
• Add Datastream to Spanner IT for PostgreSQL inheritance tables by @jsuhani-2026 in #4082
• test(spanner): add tests for creating tables w/o primary keys by @hengfengli in #4073
• [Data Validation] Add GCSSpannerDVSchemaTransformationsIT by @aasthabharill in #4080
• [Data Validation] Add Wide Row IT for Max PK Columns by @aasthabharill in #4090
• [Data Validation] Add wide row tests for max string length by @aasthabharill in #4094
• [Data Validation] Add wide row tests for 10 MiB cell and 8KiB PK by @aasthabharill in #4092

Bug fixes

• build: update to latest import-spec version by @fbiville in #4084

Improvements

• refactor: simplify template logic by @fbiville in #3582
• update gsutils ref by @derrickaw in #4098

Full Changelog: 2026-07-28-00_RC00...2026-08-04-00_RC00

Dataflow Templates 2026-07-28-00_RC00

What's Changed

Additional New Features

• [Data Validation] Add Integration Testing Framework and Core Matching IT by @aasthabharill in #4031
• [Data Validation] Add GCSSpannerDvRenamedColumnsIT and GCSSpannerDvRenamedTableIT by @aasthabharill in #4055
• Adding support for custom data generation using JAR by @shreyakhajanchi in #4024
• [Data Validation] Add GCSSpannerDvSchemaIT by @aasthabharill in #4054
• [Data Validation] First E2E Integration Test + Base class by @aasthabharill in #4061
• [Data Validation] Add Sharded Setup IT by @aasthabharill in #4067

Bug fixes

• try fixing missing delta lake read class by @derrickaw in #4038
• Run Validation ITs using DirectRunner by @aasthabharill in #4057

Improvements

• update package version by @derrickaw in #4044

New Contributors

• @yonatandt made their first contribution in #4039

Full Changelog: 2026-07-21-00_RC00...2026-07-28-00_RC00