Bump github.com/apache/beam/sdks/v2 from 2.49.0 to 2.50.0 #905

dependabot · 2023-08-31T04:27:44Z

Bumps github.com/apache/beam/sdks/v2 from 2.49.0 to 2.50.0.

Release notes

Sourced from github.com/apache/beam/sdks/v2's releases.

Beam 2.50.0 release

We are happy to present the new 2.50.0 release of Beam. This release includes both improvements and new functionality. See the download page for this release.

For more information on changes in 2.50.0, check out the detailed release notes.

Highlights

Spark 3.2.2 is used as default version for Spark runner (#23804).

The Go SDK has a new default local runner, called Prism (#24789).

All Beam released container images are now multi-arch images that support both x86 and ARM CPU architectures.

I/Os

Java KafkaIO now supports picking up topics via topicPattern (#26948)

Support for read from Cosmos DB Core SQL API (#23604)

Upgraded to HBase 2.5.5 for HBaseIO. (Java) (#27711)

Added support for GoogleAdsIO source (Java) (#27681).

New Features / Improvements

The Go SDK now requires Go 1.20 to build. (#27558)

The Go SDK has a new default local runner, Prism. (#24789).

Prism is a portable runner that executes each transform independantly, ensuring coders.

At this point it supercedes the Go direct runner in functionality. The Go direct runner is now deprecated.

See https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/runners/prism/README.md for the goals and features of Prism.

Hugging Face Model Handler for RunInference added to Python SDK. (#26632)

Hugging Face Pipelines support for RunInference added to Python SDK. (#27399)

Vertex AI Model Handler for RunInference now supports private endpoints (#27696)

MLTransform transform added with support for common ML pre/postprocessing operations (#26795)

Upgraded the Kryo extension for the Java SDK to Kryo 5.5.0. This brings in bug fixes, performance improvements, and serialization of Java 14 records. (#27635)

All Beam released container images are now multi-arch images that support both x86 and ARM CPU architectures. (#27674). The multi-arch container images include:

All versions of Go, Python, Java and Typescript SDK containers.

All versions of Flink job server containers.

Java and Python expansion service containers.

Transform service controller container.

Spark3 job server container.

Added support for batched writes to AWS SQS for improved throughput (Java, AWS 2).(#21429)

Breaking Changes

Python SDK: Legacy runner support removed from Dataflow, all pipelines must use runner v2.

Python SDK: Dataflow Runner will no longer stage Beam SDK from PyPI in the --staging_location at pipeline submission. Custom container images that are not based on Beam's default image must include Apache Beam installation.(#26996)

Deprecations

The Go Direct Runner is now Deprecated. It remains available to reduce migration churn.

... (truncated)

Changelog

Sourced from github.com/apache/beam/sdks/v2's changelog.

[2.50.0] - 2023-08-30

Highlights

Spark 3.2.2 is used as default version for Spark runner (#23804).

The Go SDK has a new default local runner, called Prism (#24789).

All Beam released container images are now multi-arch images that support both x86 and ARM CPU architectures.

I/Os

Java KafkaIO now supports picking up topics via topicPattern (#26948)

Support for read from Cosmos DB Core SQL API (#23604)

Upgraded to HBase 2.5.5 for HBaseIO. (Java) (#27711)

Added support for GoogleAdsIO source (Java) (#27681).

New Features / Improvements

The Go SDK now requires Go 1.20 to build. (#27558)

The Go SDK has a new default local runner, Prism. (#24789).

Prism is a portable runner that executes each transform independantly, ensuring coders.

At this point it supercedes the Go direct runner in functionality. The Go direct runner is now deprecated.

See https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/runners/prism/README.md for the goals and features of Prism.

Hugging Face Model Handler for RunInference added to Python SDK. (#26632)

Hugging Face Pipelines support for RunInference added to Python SDK. (#27399)

Vertex AI Model Handler for RunInference now supports private endpoints (#27696)

MLTransform transform added with support for common ML pre/postprocessing operations (#26795)

Upgraded the Kryo extension for the Java SDK to Kryo 5.5.0. This brings in bug fixes, performance improvements, and serialization of Java 14 records. (#27635)

All Beam released container images are now multi-arch images that support both x86 and ARM CPU architectures. (#27674). The multi-arch container images include:

All versions of Go, Python, Java and Typescript SDK containers.

All versions of Flink job server containers.

Java and Python expansion service containers.

Transform service controller container.

Spark3 job server container.

Added support for batched writes to AWS SQS for improved throughput (Java, AWS 2).(#21429)

Breaking Changes

Python SDK: Legacy runner support removed from Dataflow, all pipelines must use runner v2.

Python SDK: Dataflow Runner will no longer stage Beam SDK from PyPI in the --staging_location at pipeline submission. Custom container images that are not based on Beam's default image must include Apache Beam installation.(#26996)

Deprecations

The Go Direct Runner is now Deprecated. It remains available to reduce migration churn.

Tests can be set back to the direct runner by overriding TestMain: func TestMain(m *testing.M) { ptest.MainWithDefault(m, "direct") }

It's recommended to fix issues seen in tests using Prism, as they can also happen on any portable runner.

Use the generic register package for your pipeline DoFns to ensure pipelines function on portable runners, like prism.

Do not rely on closures or using package globals for DoFn configuration. They don't function on portable runners.

Bugfixes

... (truncated)

Commits

bb044f4 Set version for 2.50.0 RC2
8f762e3 Add back docker prune step between python container pushes (#28033) (#28101)
28d20e4 Loosen environment validation for RCs (#28110)
b1d245d Pin hugging face docs (#28106) (#28115)
ff12f23 [Cherry-pick #28036] Add libgeos-dev for ARM components of multi-arch Beam SD...
8293bbd [Cherry Pick #28083][prism] support single external env pipelines. (#28111)
c42daee Cherry pick #27842 (#28015)
50e2495 Revert #27878 (#27968) (#27970)
8caa7ce Set Dataflow container to release version.
47e8de3 Merge pull request #27884 Also use config pattern for yaml providers.
Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR
@dependabot recreate will recreate this PR, overwriting any edits that have been made to it
@dependabot merge will merge this PR after your CI passes on it
@dependabot squash and merge will squash and merge this PR after your CI passes on it
@dependabot cancel merge will cancel a previously requested merge and block automerging
@dependabot reopen will reopen this PR if it is closed
@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
@dependabot show <dependency name> ignore conditions will show all of the ignore conditions of the specified dependency
@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

AlCutter · 2023-09-05T10:07:25Z

:{key:"n13" value:{unique_name:"n13" coder_id:"c9" is_bounded:BOUNDED windowing_strategy_id:"w0"}} pcollections:{key:"n14" value:{unique_name:"n14" coder_id:"c0" is_bounded:BOUNDED windowing_strategy_id:"w0"}} pcollections:{key:"n15" value:{unique_name:"n15" coder_id:"c3" is_bounded:BOUNDED windowing_strategy_id:"w0"}} pcollections:{key:"n16" value:{unique_name:"n16" coder_id:"c4" is_bounded:BOUNDED windowing_strategy_id:"w0"}} pcollections:{key:"n17" value:{unique_name:"n17" coder_id:"c5" is_bounded:BOUNDED windowing_strategy_id:"w0"}} pcollections:{key:"n18" value:{unique_name:"n18" coder_id:"c0" is_bounded:BOUNDED windowing_strategy_id:"w0"}} pcollections:{key:"n19" value:{unique_name:"n19" coder_id:"c6" is_bounded:BOUNDED windowing_strategy_id:"w0"}} pcollections:{key:"n2" value:{unique_name:"n2" coder_id:"c4" is_bounded:BOUNDED windowing_strategy_id:"w0"}} pcollections:{key:"n20" value:{unique_name:"n20" coder_id:"c10" is_bounded:BOUNDED windowing_strategy_id:"w0"}} pcollections:{key:"n20_e22_inject0" value:{unique_name:"n20_e22_inject0" coder_id:"c14" is_bounded:BOUNDED windowing_strategy_id:"w0"}} pcollections:{key:"n21" value:{unique_name:"n21" coder_id:"c7" is_bounded:BOUNDED windowing_strategy_id:"w0"}} pcollections:{key:"n21_e22_inject1" value:{unique_name:"n21_e22_inject1" coder_id:"c14" is_bounded:BOUNDED windowing_strategy_id:"w0"}} pcollections:{key:"n22" value:{unique_name:"n22" coder_id:"c20" is_bounded:BOUNDED windowing_strategy_id:"w0"}} pcollections:{key:"n22_flatten" value:{unique_name:"n22_flatten" coder_id:"c14" is_bounded:BOUNDED windowing_strategy_id:"w0"}} pcollections:{key:"n22_out" value:{unique_name:"n22_out" coder_id:"c16" is_bounded:BOUNDED windowing_strategy_id:"w0"}} pcollections:{key:"n23" value:{unique_name:"n23" coder_id:"c0" is_bounded:BOUNDED windowing_strategy_id:"w0"}} pcollections:{key:"n24" value:{unique_name:"n24" coder_id:"c2" is_bounded:BOUNDED windowing_strategy_id:"w0"}} pcollections:{key:"n25" value:{unique_name:"n25" coder_id:"c3" is_bounded:BOUNDED windowing_strategy_id:"w0"}} pcollections:{key:"n26" value:{unique_name:"n26" coder_id:"c2" is_bounded:BOUNDED windowing_strategy_id:"w0"}} pcollections:{key:"n27" value:{unique_name:"n27" coder_id:"c3" is_bounded:BOUNDED windowing_strategy_id:"w0"}} pcollections:{key:"n28" value:{unique_name:"n28" coder_id:"c2" is_bounded:BOUNDED windowing_strategy_id:"w0"}} pcollections:{key:"n29" value:{unique_name:"n29" coder_id:"c2" is_bounded:BOUNDED windowing_strategy_id:"w0"}} pcollections:{key:"n3" value:{unique_name:"n3" coder_id:"c5" is_bounded:BOUNDED windowing_strategy_id:"w0"}} pcollections:{key:"n30" value:{unique_name:"n30" coder_id:"c2" is_bounded:BOUNDED windowing_strategy_id:"w0"}} pcollections:{key:"n31" value:{unique_name:"n31" coder_id:"c3" is_bounded:BOUNDED windowing_strategy_id:"w0"}} pcollections:{key:"n32" value:{unique_name:"n32" coder_id:"c2" is_bounded:BOUNDED windowing_strategy_id:"w0"}} pcollections:{key:"n33" value:{unique_name:"n33" coder_id:"c3" is_bounded:BOUNDED windowing_strategy_id:"w0"}} pcollections:{key:"n34" value:{unique_name:"n34" coder_id:"c2" is_bounded:BOUNDED windowing_strategy_id:"w0"}} pcollections:{key:"n35" value:{unique_name:"n35" coder_id:"c3" is_bounded:BOUNDED windowing_strategy_id:"w0"}} pcollections:{key:"n36" value:{unique_name:"n36" coder_id:"c2" is_bounded:BOUNDED windowing_strategy_id:"w0"}} pcollections:{key:"n37" value:{unique_name:"n37" coder_id:"c2" is_bounded:BOUNDED windowing_strategy_id:"w0"}} pcollections:{key:"n38" value:{unique_name:"n38" coder_id:"c2" is_bounded:BOUNDED windowing_strategy_id:"w0"}} pcollections:{key:"n39" value:{unique_name:"n39" coder_id:"c3" is_bounded:BOUNDED windowing_strategy_id:"w0"}} pcollections:{key:"n4" value:{unique_name:"n4" coder_id:"c6" is_bounded:BOUNDED windowing_strategy_id:"w0"}} pcollections:{key:"n5" value:{unique_name:"n5" coder_id:"c7" is_bounded:BOUNDED windowing_strategy_id:"w0"}} pcollections:{key:"n6" value:{unique_name:"n6" coder_id:"c9" is_bounded:BOUNDED windowing_strategy_id:"w0"}} pcollections:{key:"n7" value:{unique_name:"n7" coder_id:"c0" is_bounded:BOUNDED windowing_strategy_id:"w0"}} pcollections:{key:"n8" value:{unique_name:"n8" coder_id:"c3" is_bounded:BOUNDED windowing_strategy_id:"w0"}} pcollections:{key:"n9" value:{unique_name:"n9" coder_id:"c4" is_bounded:BOUNDED windowing_strategy_id:"w0"}} windowing_strategies:{key:"w0" value:{window_fn:{urn:"beam:window_fn:global_windows:v1"} merge_status:NON_MERGING window_coder_id:"c1" trigger:{default:{}} accumulation_mode:DISCARDING output_time:END_OF_WINDOW closing_behavior:EMIT_IF_NONEMPTY on_time_behavior:FIRE_IF_NONEMPTY environment_id:"go"}} coders:{key:"c0" value:{spec:{urn:"beam:coder:row:v1" payload:"\n\n\n\x04Path\x1a\x02\x10\t\n\x0e\n\x08RootHash\x1a\x02\x10\t\nR\n\x06Leaves\x1aH\x1aF\nD\x08\x012@\n>\n\n\n\x04Path\x1a\x02\x10\t\n\n\n\x04Hash\x1a\x02\x10\t\x12$e1ea475c-a7d0-4dd6-8766-bdf62c66d54f\x12$894dcbd9-a9bf-4663-83c2-af34ca1cedc3\x1a&\n\x1abeam:schema:go:nillable:v1\x12\x02\x10\x08\x1a\x04\n\x02@\x01"}}} coders:{key:"c0_lp" value:{spec:{urn:"beam:coder:length_prefix:v1"} component_coder_ids:"c0"}} coders:{key:"c1" value:{spec:{urn:"beam:coder:global_window:v1"}}} coders:{key:"c2" value:{spec:{urn:"beam:coder:string_utf8:v1"}}} coders:{key:"cwv_n24" value:{spec:{urn:"beam:coder:windowed_value:v1"} component_coder_ids:"c2" component_coder_ids:"c1"}} coders:{key:"cwv_n7" value:{spec:{urn:"beam:coder:windowed_value:v1"} component_coder_ids:"c0_lp" component_coder_ids:"c1"}} state_api_service_descriptor:{url:"[::]:33911"}
Step #5 - "presubmit-build-and-test-with-coverage": 	
Step #5 - "presubmit-build-and-test-with-coverage": 	decoding userfn fn:{fn:{name:"github.com/google/trillian-examples/experimental/batchmap/sumdb/build/pipeline.TestCreateAndUpdateEquivalence.func1.1" type:{kind:FUNC parameter_types:{kind:PTR element:{kind:EXTERNAL external_key:"github.com/google/trillian/experimental/batchmap.Tile"}} return_types:{kind:STRING}}}} opcode:"ParDo" inbound:{kind:MAIN type:{type:{kind:PTR element:{kind:EXTERNAL external_key:"github.com/google/trillian/experimental/batchmap.Tile"}}}} outbound:{type:{type:{kind:STRING}}}
Step #5 - "presubmit-build-and-test-with-coverage": bad function
Step #5 - "presubmit-build-and-test-with-coverage": 	caused by:
Step #5 - "presubmit-build-and-test-with-coverage": 	decoding DoFn fn:{name:"github.com/google/trillian-examples/experimental/batchmap/sumdb/build/pipeline.TestCreateAndUpdateEquivalence.func1.1" type:{kind:FUNC parameter_types:{kind:PTR element:{kind:EXTERNAL external_key:"github.com/google/trillian/experimental/batchmap.Tile"}} return_types:{kind:STRING}}}
Step #5 - "presubmit-build-and-test-with-coverage": failed to decode userfn
Step #5 - "presubmit-build-and-test-with-coverage": 	caused by:
Step #5 - "presubmit-build-and-test-with-coverage": github.com/google/trillian-examples/experimental/batchmap/sumdb/build/pipeline.TestCreateAndUpdateEquivalence.func1.1 not found. Register DoFns and functions with the the beam/register package. source=/go/pkg/mod/github.com/apache/beam/sdks/v2@v2.50.0/go/pkg/beam/runners/universal/runnerlib/execute.go:109 time=2023-08-31T09:15:16.190Z worker.ID=go worker.endpoint=[::]:33911
Step #5 - "presubmit-build-and-test-with-coverage": 2023/08/31 09:15:16 INFO stopping worker go source=/go/pkg/mod/github.com/apache/beam/sdks/v2@v2.50.0/go/pkg/beam/runners/universal/extworker/extworker.go:117 time=2023-08-31T09:15:16.190Z worker.ID=go worker.endpoint=[::]:33911
Step #5 - "presubmit-build-and-test-with-coverage": --- FAIL: TestCreateAndUpdateEquivalence (0.04s)
Step #5 - "presubmit-build-and-test-with-coverage":     --- FAIL: TestCreateAndUpdateEquivalence/No_logs (0.04s)
Step #5 - "presubmit-build-and-test-with-coverage":         pipeline_test.go:92: unexpected error: job job-006 failed:
Step #5 - "presubmit-build-and-test-with-coverage":             ProcessBundle failed: 	invalid bundle desc: stage038
Step #5 - "presubmit-build-and-test-with-coverage":             	id:"stage038" transforms:{key:"e24" value:{unique_name:"pipeline.TestCreateAndUpdateEquivalence.func1.1" spec:{urn:"beam:transform:pardo:v1" payload:"\n\xbe\x03\n\x19beam:go:transform:dofn:v1\x1a\xa0\x03ChliZWFtOmdvOnRyYW5zZm9ybTpkb2ZuOnYxEpgCCsEBCr4BCnVnaXRodWIuY29tL2dvb2dsZS90cmlsbGlhbi1leGFtcGxlcy9leHBlcmltZW50YWwvYmF0Y2htYXAvc3VtZGIvYnVpbGQvcGlwZWxpbmUuVGVzdENyZWF0ZUFuZFVwZGF0ZUVxdWl2YWxlbmNlLmZ1bmMxLjESRQgWIj0IGBI5CBpKNWdpdGh1Yi5jb20vZ29vZ2xlL3RyaWxsaWFuL2V4cGVyaW1lbnRhbC9iYXRjaG1hcC5UaWxlKgIIDBJDCAESPwo9CBgSOQgaSjVnaXRodWIuY29tL2dvb2dsZS90cmlsbGlhbi9leHBlcmltZW50YWwvYmF0Y2htYXAuVGlsZRoGCgQKAggMIgVQYXJEbw=="} inputs:{key:"i0" value:"n7"} outputs:{key:"i0" value:"n24"} environment_id:"go"}} transforms:{key:"e24_i0" value:{unique_name:"e24_i0" spec:{urn:"beam:runner:sink:v1" payload:"\n\x0c\n\n[::]:33911\x12\x07cwv_n24"} inputs:{key:"i0" value:"n24"}}} transforms:{key:"stage038_source" value:{unique_name:"stage038_source" spec:{urn:"beam:runner:source:v1" payload:"\n\x0c\n\n[::]:33911\x12\x06cwv_n7"} outputs:{key:"i0" value:"n7"}}} pcollections:{key:"n1" value:{unique_name:"n1" coder_id:"c3" is_bounded:BOUNDED windowing_strategy_id:"w0"}} pcollections:{key:"n10" value:{unique_name:"n10" coder_id:"c5" is_bounded:BOUNDED windowing_strategy_id:"w0"}} pcollections:{key:"n11" value:{unique_name:"n11" coder_id:"c6" is_bounded:BOUNDED windowing_strategy_id:"w0"}} pcollections:{key:"n12" value:{unique_name:"n12" coder_id:"c7" is_bounded:BOUNDED windowing_strategy_id:"w0"}} pcollections:{key:"n13" value:{unique_name:"n13" coder_id:"c9" is_bounded:BOUNDED windowing_strategy_id:"w0"}} pcollections:{key:"n14" value:{unique_name:"n14" coder_id:"c0" is_bounded:BOUNDED windowing_strategy_id:"w0"}} pcollections:{key:"n15" value:{unique_name:"n15" coder_id:"c3" is_bounded:BOUNDED windowing_strategy_id:"w0"}} pcollections:{key:"n16" value:{unique_name:"n16" coder_id:"c4" is_bounded:BOUNDED windowing_strategy_id:"w0"}} pcollections:{key:"n17" value:{unique_name:"n17" coder_id:"c5" is_bounded:BOUNDED windowing_strategy_id:"w0"}} pcollections:{key:"n18" value:{unique_name:"n18" coder_id:"c0" is_bounded:BOUNDED windowing_strategy_id:"w0"}} pcollections:{key:"n19" value:{unique_name:"n19" coder_id:"c6" is_bounded:BOUNDED windowing_strategy_id:"w0"}} pcollections:{key:"n2" value:{unique_name:"n2" coder_id:"c4" is_bounded:BOUNDED windowing_strategy_id:"w0"}} pcollections:{key:"n20" value:{unique_name:"n20" coder_id:"c10" is_bounded:BOUNDED windowing_strategy_id:"w0"}} pcollections:{key:"n20_e22_inject0" value:{unique_name:"n20_e22_inject0" coder_id:"c14" is_bounded:BOUNDED windowing_strategy_id:"w0"}} pcollections:{key:"n21" value:{unique_name:"n21" coder_id:"c7" is_bounded:BOUNDED windowing_strategy_id:"w0"}} pcollections:{key:"n21_e22_inject1" value:{unique_name:"n21_e22_inject1" coder_id:"c14" is_bounded:BOUNDED windowing_strategy_id:"w0"}} pcollections:{key:"n22" value:{unique_name:"n22" coder_id:"c20" is_bounded:BOUNDED windowing_strategy_id:"w0"}} pcollections:{key:"n22_flatten" value:{unique_name:"n22_flatten" coder_id:"c14" is_bounded:BOUNDED windowing_strategy_id:"w0"}} pcollections:{key:"n22_out" value:{unique_name:"n22_out" coder_id:"c16" is_bounded:BOUNDED windowing_strategy_id:"w0"}} pcollections:{key:"n23" value:{unique_name:"n23" coder_id:"c0" is_bounded:BOUNDED windowing_strategy_id:"w0"}} pcollections:{key:"n24" value:{unique_name:"n24" coder_id:"c2" is_bounded:BOUNDED windowing_strategy_id:"w0"}} pcollections:{key:"n25" value:{unique_name:"n25" coder_id:"c3" is_bounded:BOUNDED windowing_strategy_id:"w0"}} pcollections:{key:"n26" value:{unique_name:"n26" coder_id:"c2" is_bounded:BOUNDED windowing_strategy_id:"w0"}} pcollections:{key:"n27" value:{unique_name:"n27" coder_id:"c3" is_bounded:BOUNDED windowing_strategy_id:"w0"}} pcollections:{key:"n28" value:{unique_name:"n28" coder_id:"c2" is_bounded:BOUNDED windowing_strategy_id:"w0"}} pcollections:{key:"n29" value:{unique_name:"n29" coder_id:"c2" is_bounded:BOUNDED windowing_strategy_id:"w0"}} pcollections:{key:"n3" value:{unique_name:"n3" coder_id:"c5" is_bounded:BOUNDED windowing_strategy_id:"w0"}} pcollections:{key:"n30" value:{unique_name:"n30" coder_id:"c2" is_bounded:BOUNDED windowing_strategy_id:"w0"}} pcollections:{key:"n31" value:{unique_name:"n31" coder_id:"c3" is_bounded:BOUNDED windowing_strategy_id:"w0"}} pcollections:{key:"n32" value:{unique_name:"n32" coder_id:"c2" is_bounded:BOUNDED windowing_strategy_id:"w0"}} pcollections:{key:"n33" value:{unique_name:"n33" coder_id:"c3" is_bounded:BOUNDED windowing_strategy_id:"w0"}} pcollections:{key:"n34" value:{unique_name:"n34" coder_id:"c2" is_bounded:BOUNDED windowing_strategy_id:"w0"}} pcollections:{key:"n35" value:{unique_name:"n35" coder_id:"c3" is_bounded:BOUNDED windowing_strategy_id:"w0"}} pcollections:{key:"n36" value:{unique_name:"n36" coder_id:"c2" is_bounded:BOUNDED windowing_strategy_id:"w0"}} pcollections:{key:"n37" value:{unique_name:"n37" coder_id:"c2" is_bounded:BOUNDED windowing_strategy_id:"w0"}} pcollections:{key:"n38" value:{unique_name:"n38" coder_id:"c2" is_bounded:BOUNDED windowing_strategy_id:"w0"}} pcollections:{key:"n39" value:{unique_name:"n39" coder_id:"c3" is_bounded:BOUNDED windowing_strategy_id:"w0"}} pcollections:{key:"n4" value:{unique_name:"n4" coder_id:"c6" is_bounded:BOUNDED windowing_strategy_id:"w0"}} pcollections:{key:"n5" value:{unique_name:"n5" coder_id:"c7" is_bounded:BOUNDED windowing_strategy_id:"w0"}} pcollections:{key:"n6" value:{unique_name:"n6" coder_id:"c9" is_bounded:BOUNDED windowing_strategy_id:"w0"}} pcollections:{key:"n7" value:{unique_name:"n7" coder_id:"c0" is_bounded:BOUNDED windowing_strategy_id:"w0"}} pcollections:{key:"n8" value:{unique_name:"n8" coder_id:"c3" is_bounded:BOUNDED windowing_strategy_id:"w0"}} pcollections:{key:"n9" value:{unique_name:"n9" coder_id:"c4" is_bounded:BOUNDED windowing_strategy_id:"w0"}} windowing_strategies:{key:"w0" value:{window_fn:{urn:"beam:window_fn:global_windows:v1"} merge_status:NON_MERGING window_coder_id:"c1" trigger:{default:{}} accumulation_mode:DISCARDING output_time:END_OF_WINDOW closing_behavior:EMIT_IF_NONEMPTY on_time_behavior:FIRE_IF_NONEMPTY environment_id:"go"}} coders:{key:"c0" value:{spec:{urn:"beam:coder:row:v1" payload:"\n\n\n\x04Path\x1a\x02\x10\t\n\x0e\n\x08RootHash\x1a\x02\x10\t\nR\n\x06Leaves\x1aH\x1aF\nD\x08\x012@\n>\n\n\n\x04Path\x1a\x02\x10\t\n\n\n\x04Hash\x1a\x02\x10\t\x12$e1ea475c-a7d0-4dd6-8766-bdf62c66d54f\x12$894dcbd9-a9bf-4663-83c2-af34ca1cedc3\x1a&\n\x1abeam:schema:go:nillable:v1\x12\x02\x10\x08\x1a\x04\n\x02@\x01"}}} coders:{key:"c0_lp" value:{spec:{urn:"beam:coder:length_prefix:v1"} component_coder_ids:"c0"}} coders:{key:"c1" value:{spec:{urn:"beam:coder:global_window:v1"}}} coders:{key:"c2" value:{spec:{urn:"beam:coder:string_utf8:v1"}}} coders:{key:"cwv_n24" value:{spec:{urn:"beam:coder:windowed_value:v1"} component_coder_ids:"c2" component_coder_ids:"c1"}} coders:{key:"cwv_n7" value:{spec:{urn:"beam:coder:windowed_value:v1"} component_coder_ids:"c0_lp" component_coder_ids:"c1"}} state_api_service_descriptor:{url:"[::]:33911"}
Step #5 - "presubmit-build-and-test-with-coverage":             	
Step #5 - "presubmit-build-and-test-with-coverage":             	decoding userfn fn:{fn:{name:"github.com/google/trillian-examples/experimental/batchmap/sumdb/build/pipeline.TestCreateAndUpdateEquivalence.func1.1" type:{kind:FUNC parameter_types:{kind:PTR element:{kind:EXTERNAL external_key:"github.com/google/trillian/experimental/batchmap.Tile"}} return_types:{kind:STRING}}}} opcode:"ParDo" inbound:{kind:MAIN type:{type:{kind:PTR element:{kind:EXTERNAL external_key:"github.com/google/trillian/experimental/batchmap.Tile"}}}} outbound:{type:{type:{kind:STRING}}}
Step #5 - "presubmit-build-and-test-with-coverage":             bad function
Step #5 - "presubmit-build-and-test-with-coverage":             	caused by:
Step #5 - "presubmit-build-and-test-with-coverage":             	decoding DoFn fn:{name:"github.com/google/trillian-examples/experimental/batchmap/sumdb/build/pipeline.TestCreateAndUpdateEquivalence.func1.1" type:{kind:FUNC parameter_types:{kind:PTR element:{kind:EXTERNAL external_key:"github.com/google/trillian/experimental/batchmap.Tile"}} return_types:{kind:STRING}}}
Step #5 - "presubmit-build-and-test-with-coverage":             failed to decode userfn
Step #5 - "presubmit-build-and-test-with-coverage":             	caused by:
Step #5 - "presubmit-build-and-test-with-coverage":             github.com/google/trillian-examples/experimental/batchmap/sumdb/build/pipeline.TestCreateAndUpdateEquivalence.func1.1 not found. Register DoFns and functions with the the beam/register package.
Step #5 - "presubmit-build-and-test-with-coverage":             error may be due to Apache Beam Go's migration from the direct runner to the prism runner. While the failure(s) should be fixed, you can continue to use the direct runner with this TestMain override: `func TestMain(m *testing.M) { ptest.MainWithDefault(m, "direct") }`
Step #5 - "presubmit-build-and-test-with-coverage": FAIL
Step #5 - "presubmit-build-and-test-with-coverage": 	github.com/google/trillian-examples/experimental/batchmap/sumdb/build/pipeline	coverage: 50.0% of statements

roger2hk · 2023-09-05T10:46:08Z

Got to bump Go version to 1.20 due to apache/beam#27558. I will create a PR.

AlCutter · 2023-09-05T10:57:16Z

There are some code changes req'd for the beam update to work, I've started on them in google/trillian#3083 and I'll do the same here once that PR is in.

AlCutter · 2023-09-05T12:55:55Z

@dependabot rebase

AlCutter · 2023-09-05T13:00:13Z

/gcbrun

AlCutter · 2023-09-05T14:24:22Z

@dependabot rebase

AlCutter · 2023-09-05T14:26:10Z

/gcbrun

AlCutter · 2023-09-05T15:45:37Z

@dependabot rebase

AlCutter · 2023-09-05T15:47:44Z

/gcbrun

AlCutter · 2023-09-05T16:07:18Z

/gcbrun

Bumps [github.com/apache/beam/sdks/v2](https://github.com/apache/beam) from 2.49.0 to 2.50.0. - [Release notes](https://github.com/apache/beam/releases) - [Changelog](https://github.com/apache/beam/blob/master/CHANGES.md) - [Commits](apache/beam@v2.49.0...v2.50.0) --- updated-dependencies: - dependency-name: github.com/apache/beam/sdks/v2 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>

AlCutter · 2023-09-07T13:34:45Z

/gcbrun

AlCutter · 2023-09-07T13:40:47Z

/gcbrun

dependabot bot requested a review from a team as a code owner August 31, 2023 04:27

dependabot bot requested review from AlCutter and removed request for a team August 31, 2023 04:27

dependabot bot added dependencies Pull requests that update a dependency file go Pull requests that update Go code labels Aug 31, 2023

roger2hk approved these changes Aug 31, 2023

View reviewed changes

dependabot bot force-pushed the dependabot/go_modules/github.com/apache/beam/sdks/v2-2.50.0 branch from b921dd0 to c7e6036 Compare September 5, 2023 11:23

dependabot bot force-pushed the dependabot/go_modules/github.com/apache/beam/sdks/v2-2.50.0 branch from c7e6036 to e1b3b8b Compare September 5, 2023 12:58

dependabot bot force-pushed the dependabot/go_modules/github.com/apache/beam/sdks/v2-2.50.0 branch from e1b3b8b to e991413 Compare September 5, 2023 14:25

dependabot bot force-pushed the dependabot/go_modules/github.com/apache/beam/sdks/v2-2.50.0 branch from e991413 to e244c48 Compare September 5, 2023 15:46

dependabot bot and others added 2 commits September 7, 2023 14:01

Fix up beam registrations

a3dbebe

AlCutter force-pushed the dependabot/go_modules/github.com/apache/beam/sdks/v2-2.50.0 branch from 2e373c2 to a3dbebe Compare September 7, 2023 13:01

mod tidy

888eba1

AlCutter merged commit 5cdefef into master Sep 7, 2023
5 checks passed

AlCutter deleted the dependabot/go_modules/github.com/apache/beam/sdks/v2-2.50.0 branch September 7, 2023 13:49

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bump github.com/apache/beam/sdks/v2 from 2.49.0 to 2.50.0 #905

Bump github.com/apache/beam/sdks/v2 from 2.49.0 to 2.50.0 #905

dependabot bot commented on behalf of github Aug 31, 2023 •

edited

Loading

AlCutter commented Sep 5, 2023

roger2hk commented Sep 5, 2023

AlCutter commented Sep 5, 2023

AlCutter commented Sep 5, 2023

AlCutter commented Sep 5, 2023

AlCutter commented Sep 5, 2023

AlCutter commented Sep 5, 2023

AlCutter commented Sep 5, 2023

AlCutter commented Sep 5, 2023

AlCutter commented Sep 5, 2023

AlCutter commented Sep 7, 2023

AlCutter commented Sep 7, 2023

Bump github.com/apache/beam/sdks/v2 from 2.49.0 to 2.50.0 #905

Bump github.com/apache/beam/sdks/v2 from 2.49.0 to 2.50.0 #905

Conversation

dependabot bot commented on behalf of github Aug 31, 2023 • edited Loading

Beam 2.50.0 release

Highlights

I/Os

New Features / Improvements

Breaking Changes

Deprecations

[2.50.0] - 2023-08-30

Highlights

I/Os

New Features / Improvements

Breaking Changes

Deprecations

Bugfixes

AlCutter commented Sep 5, 2023

roger2hk commented Sep 5, 2023

AlCutter commented Sep 5, 2023

AlCutter commented Sep 5, 2023

AlCutter commented Sep 5, 2023

AlCutter commented Sep 5, 2023

AlCutter commented Sep 5, 2023

AlCutter commented Sep 5, 2023

AlCutter commented Sep 5, 2023

AlCutter commented Sep 5, 2023

AlCutter commented Sep 7, 2023

AlCutter commented Sep 7, 2023

dependabot bot commented on behalf of github Aug 31, 2023 •

edited

Loading