37 Pull requests merged by 18 people

• Fix PostCommit Java DataflowV2

  #34209 merged Mar 8, 2025

• Use individual windows rather than window sets in the combining table.

  #34193 merged Mar 7, 2025

• Remove a note about cloudpickle support being experimental

  #34200 merged Mar 7, 2025

• Bump dataflow java container version to beam-master-20250307

  #34211 merged Mar 7, 2025

• Clarify BigQuery InsertRetryPolicy behavior for non-200 responses

  #34118 merged Mar 6, 2025

• Fix test

  #34162 merged Mar 6, 2025

• Fix failing test: Increase timeout

  #34181 merged Mar 6, 2025

• Fix failing test: Add more time for grpc cleanup

  #34180 merged Mar 6, 2025

• Remove retry count assertion from testInvalidRecordReceived

  #34192 merged Mar 6, 2025

• Add test ensuring pipeline options observes kwargs.

  #34177 merged Mar 6, 2025

• Catch union-of-iterables case in get_yielded_type()

  #34186 merged Mar 5, 2025

• Alloy language connector

  #34156 merged Mar 5, 2025

• Correctly parse labels if they are passed as a single string instead of a list

  #34183 merged Mar 5, 2025

• Add AlloyDB embeddings colab.

  #34184 merged Mar 5, 2025

• Fix cleanPython

  #34185 merged Mar 5, 2025

• Add histogram to metrics container

  #33043 merged Mar 5, 2025

• Install docker compose in PostCommit Python Xlang Gcp Direct to run xlang_jdbc tests in xlang direct workflow.

  #34182 merged Mar 5, 2025

• [Spark] Skip unused outputs of ParDo in SparkRunner (#33771)

  #33772 merged Mar 5, 2025

• Updates Managed Javadocs and pydocs to refer to runner specific features

  #34072 merged Mar 4, 2025

• Add Sequences support to Breaking Changes

  #34169 merged Mar 4, 2025

• fix for adding unexpected Empty Records in Nested Arrays in BigQueryIO

  #34102 merged Mar 4, 2025

• Run Python Postcommit on High mem 22. Increase max nodes and replicas

  #34170 merged Mar 4, 2025

• [Java] Fix UnboundedReaderAsSdfFn to avoid using unstarted unbounded reader.

  #34146 merged Mar 4, 2025

• Enable kafka metrics by default for streaming dataflow jobs on v1

  #34153 merged Mar 4, 2025

• Add sleep to give enough time for server to be up

  #34133 merged Mar 4, 2025

• [Java] Allow users to specify GCS custom audit entries in pipeline options

  #34134 merged Mar 4, 2025

• updated Go to 1.24.0

  #34163 merged Mar 4, 2025

• Fix hadoop version tests.

  #34155 merged Mar 4, 2025

• Add docs about withQueryFn, logic to detect other functions, and new …

  #34127 merged Mar 3, 2025

• Add explicit schema support to JdbcIO read and xlang transform.

  #34128 merged Mar 3, 2025

• Spark Runner : Replace queueStream with custom DStream in Spark streaming Flatten transform

  #34080 merged Mar 3, 2025

• Use bigdataoss 3.x-compatible API in BigQueryIO's BatchLoads

  #34105 merged Mar 3, 2025

• Add resource hint capabilities to YAML.

  #34087 merged Mar 3, 2025

• Update republish workflow to split docker pushes

  #34086 merged Mar 3, 2025

• Make pydoc docstring reflecting deprecated

  #34136 merged Mar 3, 2025

• Call out race condition fix in CHANGES

  #34147 merged Mar 3, 2025

• Automatically refresh Performance Metrics Graphs using Looker

  #34097 merged Mar 3, 2025

35 Pull requests opened by 20 people

• [Java] Added Metrics Configuration Support to Iceberg Data Writers

  #34140 opened Mar 2, 2025

• Clean up GCP Resources (Pubsub)

  #34141 opened Mar 2, 2025

• Fix Docker dev environment set up

  #34142 opened Mar 2, 2025

• Fix Docker build error by adding fallback for python3.12-distutils

  #34144 opened Mar 3, 2025

• test commit

  #34148 opened Mar 3, 2025

• Don't attempt to cache the bulk multimap lookup call.

  #34150 opened Mar 3, 2025

• Updates the information regarding Managed I/O

  #34152 opened Mar 4, 2025

• introduce flags for customizing standard providers using their own YA…

  #34158 opened Mar 4, 2025

• Putting the default retry settings which was overridden in Dataflow templates

  #34161 opened Mar 4, 2025

• [KafkaIO] Fix average record size data race and backlog estimation

  #34165 opened Mar 4, 2025

• [Java] Added Part Spec when creating Tables

  #34166 opened Mar 4, 2025

• Bump @octokit/plugin-paginate-rest and @octokit/rest in /scripts/ci/issue-report

  #34167 opened Mar 4, 2025

• Bump github.com/aws/aws-sdk-go-v2/feature/s3/manager from 1.17.62 to 1.17.65 in /sdks

  #34179 opened Mar 5, 2025

• Bump jinja2 from 3.1.4 to 3.1.6 in /.test-infra/jenkins/metrics_report

  #34189 opened Mar 6, 2025

• Bump golang.org/x/oauth2 from 0.26.0 to 0.28.0 in /sdks

  #34190 opened Mar 6, 2025

• Rethrowing Exception from CassandraIO's ReadFn

  #34191 opened Mar 6, 2025

• Fix ProtoCoder NoSuchMethodException

  #34194 opened Mar 6, 2025

• [DO NOT MERGE] Removed < 1.66 for grpcio

  #34196 opened Mar 6, 2025

• Update pinned kafka version to 3.9.0 in Expansion Service.

  #34197 opened Mar 6, 2025

• [KafkaIO] Remove duplicate offset in range check

  #34201 opened Mar 7, 2025

• [KafkaIO] Update tracker and watermark for non-visible progress

  #34202 opened Mar 7, 2025

• Bump google.golang.org/api from 0.221.0 to 0.224.0 in /sdks

  #34203 opened Mar 7, 2025

• Add Documentation Hint for Template Job Creation in DataflowRunner

  #34204 opened Mar 7, 2025

• Add support for top-level table properties table creation

  #34205 opened Mar 7, 2025

• [WIP][Python] File staging to user worker support

  #34208 opened Mar 7, 2025

• [Java] Add parsedData to Hl7v2Message and Update HL7v2IO Docs

  #34213 opened Mar 7, 2025

• Declare ExecutionStateTracker.nextBundleLullDurationReportMs as volitile

  #34214 opened Mar 7, 2025

• Issue warning on long running DoFn.Setup on legacy Dataflow runner

  #34215 opened Mar 7, 2025

• Fix typo on resource hints page.

  #34216 opened Mar 8, 2025

• Add yaml examples generation to the release scripts.

  #34217 opened Mar 8, 2025

• [AnomalyDetection] Add transforms and detectors.

  #34218 opened Mar 8, 2025

• [Java] Added tests for S3ReadableSeekableByteChannel

  #34219 opened Mar 8, 2025

• [Java] Ensure Pipeline Execution Requires Configuration Options or Logs Warning

  #34220 opened Mar 8, 2025

• [Java] Add InsertRetryPolicy for non-successful BigQuery insertAll responses

  #34222 opened Mar 8, 2025

• Enable cloudpickle default

  #34223 opened Mar 8, 2025

25 Issues closed by 7 people

• [Failing Test]: org.apache.beam.runners.flink.translation.wrappers.streaming.io.source.unbounded.FlinkUnboundedSourceReaderTest.testIsAvailableAlwaysWakenUp

  #34174 closed Mar 9, 2025

• The PostCommit Java Dataflow V2 job is flaky

  #30729 closed Mar 8, 2025

• Performance Regression or Improvement: cogbk_python_batch_load_test_reiterate_4times_2MB_values:runtime

  #34187 closed Mar 6, 2025

• [Failing Test]: org.apache.beam.sdk.io.gcp.spanner.changestreams.SpannerChangeStreamErrorTest.testInvalidRecordReceived

  #34173 closed Mar 6, 2025

• [Bug]: `gradlew clean` involves external transform generation and fails

  #30954 closed Mar 5, 2025

• [Bug]: Unexpected Empty Records in Nested Arrays When Using BigQueryIO.write() with .withAutoSchemaUpdate(true) and .ignoreUnknownValues()

  #33842 closed Mar 4, 2025

• The PostCommit Java Hadoop Versions job is flaky

  #33252 closed Mar 4, 2025

• Add documentation and improved errors for QueryFn in MongoDbIO

  #21005 closed Mar 4, 2025

• Add support for checkpointing in Spark streaming

  #20426 closed Mar 3, 2025

• Flatten of Bounded and Unbounded repeats the union with the RDD for each micro-batch.

  #18144 closed Mar 3, 2025

• Performance Regression or Improvement: pytorch_image_classification_benchmarks-resnet152-mean_inference_batch_latency_micro_secs:mean_inference_batch_latency_micro_secs

  #34108 closed Mar 3, 2025

• Performance Regression or Improvement: gbk_python_batch_load_test_fanout_8_times_with_2GB_10byte_records_total:runtime

  #34143 closed Mar 3, 2025

• Performance Regression or Improvement: combine_python_batch_2gb_10_byte_records:runtime

  #34139 closed Mar 3, 2025

• Performance Regression or Improvement: test_cloudml_benchmark_criteo_10GB-runtime_sec:runtime_sec

  #34138 closed Mar 3, 2025

• Performance Regression or Improvement: cogbk_python_batch_load_test_reiterate_4times_10KB_values:runtime

  #34131 closed Mar 3, 2025

• Performance Regression or Improvement: gbk_python_batch_load_test_2gb_of_100B_records:runtime

  #34130 closed Mar 3, 2025

• Performance Regression or Improvement: gbk_python_batch_load_test_2gb_of_10B_records:runtime

  #34129 closed Mar 3, 2025

• Performance Regression or Improvement: cogbk_python_batch_load_test_reiterate_4times_2MB_values:runtime

  #34110 closed Mar 3, 2025

• Performance Regression or Improvement: cogbk_python_batch_load_test_2GB_of_100B_records_with_a_multiple_key:runtime

  #34109 closed Mar 3, 2025

• The PostCommit Java ValidatesRunner SparkStructuredStreaming job is flaky

  #30516 closed Mar 3, 2025

• The PostCommit Java ValidatesRunner Spark Java8 job is flaky

  #34126 closed Mar 3, 2025

• The PostCommit Java ValidatesRunner Spark job is flaky

  #34124 closed Mar 3, 2025

• The PostCommit Java Examples Spark job is flaky

  #34125 closed Mar 3, 2025

• [Task]: Speed up Docker Push Steps

  #34084 closed Mar 3, 2025

• [Task]: Make pydoc docstring reflecting deprecated and experimental API

  #22265 closed Mar 3, 2025

19 Issues opened by 13 people

• [Bug]: Python Unit Tests are flaky on windows

  #34221 opened Mar 8, 2025

• [Task]: Add Python AfterSynchronizedProcessingTime trigger and add an Iceberg CDC streaming read test

  #34212 opened Mar 7, 2025

• The PostCommit Java PVR Spark3 Streaming job is flaky

  #34207 opened Mar 7, 2025

• [Bug]: Long running DoFn.Setup methods lead to job failure in Dataflow Java legacy runner.

  #34206 opened Mar 7, 2025

• Performance Regression or Improvement: sideinpts_python_batch_10gb_1kb_10workers_1000window_first_iterable:runtime

  #34198 opened Mar 6, 2025

• [Feature Request]: Unify metrics that are to be aggregated on google worker and on the portable runner

  #34195 opened Mar 6, 2025

• [Bug]: `beam.io.WriteToCsv` ignores `num_shards` argument.

  #34188 opened Mar 6, 2025

• [Bug]: FlinkRunner never calls finish_bundle and OOM eventually

  #34178 opened Mar 5, 2025

• [Bug]: Provisioning runners fails

  #34176 opened Mar 4, 2025

• [Failing Test]: org.apache.beam.sdk.io.mqtt.MqttIOTest.testReadWithMetadata

  #34175 opened Mar 4, 2025

• [Failing Test]: org.apache.beam.sdk.transforms.RedistributeTest.testRedistributeAfterFixedWindows

  #34172 opened Mar 4, 2025

• [Failing Test]: org.apache.beam.runners.dataflow.worker.streaming.harness.FanOutStreamingEngineWorkerHarnessTest.testOnNewWorkerMetadata_redistributesBudget

  #34171 opened Mar 4, 2025

• [Task]: Close feature gaps between regular and CDC Iceberg sources

  #34168 opened Mar 4, 2025

• [Bug]: cassandraIO ReadAll does not let a pipeline handle or retry exceptions

  #34160 opened Mar 4, 2025

• [Feature Request]: Add ClickHouse Resource Manager for Integration Tests

  #34159 opened Mar 4, 2025

• [Feature Request]: Add InsertRetryPolicy for non-successful BigQuery insertAll responses.

  #34154 opened Mar 4, 2025

• [Bug]: Unable to use S3 bucket for ReadFromSnowflake staging bucket name

  #34151 opened Mar 3, 2025

• [Feature Request]: Properly incorporate bulk multimap side input reads into caching.

  #34149 opened Mar 3, 2025

• [Bug]: BigQueryIO - unknown repeated fields are merged incorrectly to payload

  #34145 opened Mar 3, 2025

54 Unresolved conversations

Sometimes conversations happen on old items that aren’t yet closed. Here is a list of all the Issues and Pull Requests with unresolved conversations.

• [Python] Add caching for BigQuery table definitions

  #34135 commented on Mar 9, 2025 • 45 new comments

• [Managed Iceberg] unbounded source

  #33504 commented on Mar 7, 2025 • 32 new comments

• Kafka add counters v1 uw2

  #33503 commented on Mar 7, 2025 • 15 new comments

• Use BoundedTrie metric to track lineage in IO

  #33891 commented on Mar 4, 2025 • 4 new comments

• [BEAM-6394] Add support to write protobuf data using ProtoParquetReader

  #34063 commented on Mar 7, 2025 • 2 new comments

• Switch to use registerFileSystemsOnce for SerializablePipelineOptions

  #34028 commented on Mar 4, 2025 • 1 new comment

• Add Encryption When Writing to Iceberg Tables in RecordWriter.java

  #34021 commented on Mar 7, 2025 • 1 new comment

• [Bug]: Cross-language JDBC (MSSQL) - incorrect negative Integer type conversion

  #34089 commented on Mar 8, 2025 • 0 new comments

• The StressTests Java BigQueryIO job is flaky

  #31968 commented on Mar 9, 2025 • 0 new comments

• Replace StorageV1 client with GCS client - Draft

  #28733 commented on Mar 4, 2025 • 0 new comments

• Support writing to Pubsub with ordering key; Add PubsubMessage SchemaCoder

  #31608 commented on Mar 7, 2025 • 0 new comments

• BigQueryIO uniformize direct and export reads

  #32360 commented on Mar 7, 2025 • 0 new comments

• Add portable Mqtt source and sink transforms

  #32385 commented on Mar 7, 2025 • 0 new comments

• add generics support to AutoValueUtils helpers

  #32977 commented on Mar 7, 2025 • 0 new comments

• Read RabbitMQ messages with headers containing nested objects

  #33072 commented on Mar 7, 2025 • 0 new comments

• Tour of Beam: update GroupByKey example

  #33242 commented on Mar 7, 2025 • 0 new comments

• chore: update to use versions for LTS 8

  #33451 commented on Mar 4, 2025 • 0 new comments

• Allow declaration of external dependencies for YAML UDFs.

  #34073 commented on Mar 8, 2025 • 0 new comments

• Fix incorrect nullness in FlinkJobInvoker and JobInvoker

  #33713 commented on Mar 7, 2025 • 0 new comments

• add default port for HostAndPort instances used in Windmill

  #34061 commented on Mar 7, 2025 • 0 new comments

• SnowflakeIO: be consistent with backslash escape char

  #33948 commented on Mar 7, 2025 • 0 new comments

• Bump @octokit/request-error, @actions/github and @octokit/rest in /scripts/ci/pr-bot

  #33998 commented on Mar 6, 2025 • 0 new comments

• Add support for collections.abc.Mapping

  #34001 commented on Mar 6, 2025 • 0 new comments

• Bump serialize-javascript and mocha in /sdks/typescript

  #34012 commented on Mar 3, 2025 • 0 new comments

• #34009 avro generic record to beam row conversion added support for a…

  #34024 commented on Mar 5, 2025 • 0 new comments

• add vendor to manually shutdown and restart GetWorkerMetadataStream to prevent DEADLINE_EXCEEDED errors

  #34053 commented on Mar 7, 2025 • 0 new comments

• add equals hashCode to BoundedToUnboundedSourceAdapter

  #34057 commented on Mar 5, 2025 • 0 new comments

• [Feature Request]: Integrate Apache Beam with Open Lineage

  #33981 commented on Mar 2, 2025 • 0 new comments

• [Bug]: Python JDBC IO Try To Connect RDB Before Deploying

  #23029 commented on Mar 3, 2025 • 0 new comments

• Beam metrics should be displayed in Flink UI "Metrics" tab

  #20691 commented on Mar 3, 2025 • 0 new comments

• [Feature Request]: [IcebergIO] Allow users to specify a partition spec when creating tables

  #34117 commented on Mar 4, 2025 • 0 new comments

• [Bug]: gprcio limitation to < 1.66 in Python is problematic

  #34081 commented on Mar 4, 2025 • 0 new comments

• [Feature Request]: Apply encryption when writing to Iceberg

  #33986 commented on Mar 4, 2025 • 0 new comments

• [Feature Request]: Upgrade to Iceberg >1.7.0 and support timestamp nano types

  #34098 commented on Mar 4, 2025 • 0 new comments

• [Feature Request]: [IcebergIO] Allow users to pass table properties to be set when creating a table

  #34116 commented on Mar 4, 2025 • 0 new comments

• [Bug]: Race condition in FileSystems initialisation

  #33965 commented on Mar 5, 2025 • 0 new comments

• The PostCommit Java IO Performance Tests job is flaky

  #30527 commented on Mar 5, 2025 • 0 new comments

• [Task]: Update the minor version of cloudpickle library prior to Beam release.

  #23119 commented on Mar 5, 2025 • 0 new comments

• The IcebergIO Integration Tests job is flaky

  #31931 commented on Mar 5, 2025 • 0 new comments

• [Bug]: Iceberg sink is not resilient to worker crash

  #34074 commented on Mar 5, 2025 • 0 new comments

• [Feature Request]: [IcebergIO] Configure data writers to track metrics

  #34112 commented on Mar 5, 2025 • 0 new comments

• [Task]: Create a script to train sklearn model for IT test.

  #24903 commented on Mar 5, 2025 • 0 new comments

• The PostCommit Java ValidatesRunner Flink Java8 job is flaky

  #32949 commented on Mar 5, 2025 • 0 new comments

• [Bug]: java.lang.IllegalStateException: Expected output stream to be empty

  #31914 commented on Mar 6, 2025 • 0 new comments

• The Go tests job is flaky

  #32627 commented on Mar 6, 2025 • 0 new comments

• [Failing Test]: dataflow runner worker project test stuck causing Java PreCommit time out

  #28957 commented on Mar 6, 2025 • 0 new comments

• Hint to template job creation in DataflowRunner / DataflowPipelineOptions

  #18217 commented on Mar 6, 2025 • 0 new comments

• [Bug]: JDBC javasdk_date:v1 decode error

  #33442 commented on Mar 6, 2025 • 0 new comments

• [Feature Request]: Upgrade or provide ETA on dill

  #22893 commented on Mar 6, 2025 • 0 new comments

• [Bug]: class org.apache.logging.slf4j.SLF4JLoggerContext cannot be cast to class org.apache.logging.log4j.core.LoggerContext (org.apache.logging.slf4j.SLF4JLoggerContext and org.apache.logging.log4j.core.LoggerContext are in unnamed module of loader 'app')

  #33983 commented on Mar 7, 2025 • 0 new comments

• [Bug]: Cross-language pipeline options are not picked up in Java DoFns

  #33074 commented on Mar 7, 2025 • 0 new comments

• [Bug]: The submission_environment_dependencies.txt file does not get staged when running with Flink runner on Dataproc

  #32743 commented on Mar 7, 2025 • 0 new comments

• The PostCommit Python job is flaky

  #30513 commented on Mar 7, 2025 • 0 new comments

• [Bug]: :sdks:python:test-suites:tox:py312:testPython312 could be flaky

  #33697 commented on Mar 8, 2025 • 0 new comments