Remove Beam Dependency Report code #26954

jrmccluskey · 2023-05-31T19:31:40Z

Deletes the Beam Dependency Report Jenkins job and associated code.

Fixes #26896

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

jrmccluskey · 2023-06-01T17:10:24Z

R: @damccorm

Is this everything that needs to be done to remove the job? As far as the python code I believe all of the references to it are in the script

github-actions · 2023-06-01T17:11:34Z

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control

damccorm

Mostly LGTM, could you please remove the reference from the README as well? https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md

jrmccluskey · 2023-06-01T17:43:37Z

Done

damccorm · 2023-06-01T17:44:35Z

.test-infra/jenkins/job_Dependency_Check.groovy

-  steps {
-    gradle {
-      rootBuildScriptDir(commonJobProperties.checkoutDir)
-      tasks('runBeamDependencyCheck')


Actually, I think we need to rip out this piece too.

This has a gradle task

beam/build.gradle.kts

Line 583 in 4f07cda

tasks.register("runBeamDependencyCheck") {

which then calls into the python sdk dependency check code -

beam/sdks/python/scripts/run_dependency_check.sh

Line 22 in 4f07cda

mkdir -p $WORKSPACE/src/build/dependencyUpdates

There's some more references that will need cleaned up around this, https://github.com/search?q=repo%3Aapache%2Fbeam%20dependencyUpdates&type=code is a good starting point

Sorry, missed that initially

I think I knocked them all out by checking dependecyCheck, dependency-check, and dependencyUpdate references along with the two direct references to the shell script itself. Fortunately touching things within the sdks/ folder triggers more of the automated testing so there will at least be an early signal of if this really breaks something

damccorm · 2023-06-01T18:05:44Z

build.gradle.kts

@@ -574,11 +574,6 @@ tasks.register("javaExamplesDataflowPrecommit") {
  dependsOn(":runners:google-cloud-dataflow-java:examples-streaming:preCommit")
 }

-tasks.register("runBeamDependencyCheck") {
-  dependsOn(":dependencyUpdates")


Looks like this comes from https://github.com/ben-manes/gradle-versions-plugin so I think we can kill

beam/build.gradle.kts

Line 25 in 8ec667c

id("com.github.ben-manes.versions") version "0.33.0"

and

beam/ownership/JAVA_DEPENDENCY_OWNERS.yaml

Line 148 in 8ec667c

group: com.github.ben-manes

Good catch on the unnecessary extra dependency

damccorm · 2023-06-01T18:07:45Z

sdks/python/tox.ini

-extras = test,gcp,aws,dataframe,interactive,interactive_test
-passenv = WORKSPACE
-commands =
-  time {toxinidir}/scripts/run_dependency_check.sh


We should be able to get rid of run_dependency_check.sh, right?

beam/sdks/python/scripts/run_dependency_check.sh

Line 4 in 8ec667c

# contributor license agreements. See the NOTICE file distributed with

Didn't realize there was another script in the Python SDK, I had already cleared out the one in the .test-infra directory

codecov · 2023-06-01T18:29:01Z

Codecov Report

Merging #26954 (1884c27) into master (d8f0e17) will increase coverage by 0.14%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #26954      +/-   ##
==========================================
+ Coverage   71.27%   71.42%   +0.14%     
==========================================
  Files         766      853      +87     
  Lines      103470   103751     +281     
==========================================
+ Hits        73747    74101     +354     
+ Misses      28184    28111      -73     
  Partials     1539     1539

Flag	Coverage Δ
python	`81.04% <ø> (+0.18%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

see 148 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

jrmccluskey · 2023-06-06T15:05:22Z

Run Python_Integration PreCommit

jrmccluskey · 2023-06-06T16:28:16Z

R: @Abacn

Re-routing this to you while Danny is out

Abacn

Thanks! this LGTM

Fix search path for the graph.dot file which contains the graph of the pipeline execution. The feature got broken after PR #26842. Adds a utility for starting up the Transform Service from Java (#26834) * Adds a utility for starting up the Transform Service from Java and adds an option for enabling it * Address reviewer comments * Addressing reviewer comments Feedback google forms iframe (#26864) * google forms * iframe * Fix tests --------- Co-authored-by: darkhan.nausharipov <darkhan.nausharipov@kzn.akvelon.com> Make lulls message in Python SDK to be consistent with the implementation in Java SDK (#26936) * Make lulls message in Python SDK to be consistent with the implementation in Java SDK * Adjust unit tests to match introduced changes * Fix linting error Revert "populate jobs with gradle scans access (#26379)" (#26972) This reverts commit dbc1c1597fef3674e6261eb8c0728735aa517a31. adding myself to author list for blog (#26984) add back deleted line + test (#26976) [GO SDK] - avroio nullable fields fix (#26812) * fixing nullable fields issue in Go avroio * readability * updating all references to goavro [#22737] Fit & Finish for Go SDK timer support. (#26782) fix output_handler (#26923) Remove this log "Default Python SDK image for environment..." (#27002) [Refactor] Beam Python Healthcare Connectors (#26917) remove 360M of installer files bloat. (#26971) Adds new python avro example, removes broken example (#26689) Add new Beam Java SDK examples (#26488) * Add new Beam Java SDK examples These examples are intended for use in interactive embedded Playground frames in the Beam documentation. * Add license comment to new examples * Fix context_line * Added example for FlatMapElements transform * Add example for Partition transform * Add example for WithKeys transform * Add CoGroupByKey example * Add examples for Combine transform * Add example for Sum transform * Add examples for Count transform * Fix message in CoGroupByKey * Add example for GroupByKey transform * Add examples for Min transform * Add example for Max transform * Use generic variants of Min transform * Add examples for Mean transform * Move new examples into examples/java * Resolve linter error * Add example for Create transform * Add example for Flatten transform * Add section names to Java WordCount examples * Improve log output in new examples * Rename examples to avoid name conflict with katas * Fix compilation of examples * Add example for Window transform * Add example for Window transform * Move new examples to the root of the examples directory as running them from new directory fails in Playground as org.apache.beam.examples.basic namespace doesn't exist in the version of Beam deployed in the Playground. * Add example for View transform * Drop "Demo" suffix from new examples * Add example for Keys transform * Remove Kata suffix * Move GroupByKey example to avoid name conflict * Add "Combiners category to CombineExample.java * Remove "multifile: true" from WindowedWordCount Removing an unnecessary dependency (#27001) update tags after including guppy dataflow legacy container (#26973) [#25896] Add an initial UI to standalone prism command. (#26961) * Fail pipelines with unimplemented win strats. * Update stateful integ tests for reuse. Validate in prism. * Make the integration tests use ptest.BuildAndRun * Make transform URN support explicit. * [#25896][prism] v-1 of jobs console ui. in progress * [prismUI] weekend work * Update styles and tidy up handlers. * consolidate error handler * pre PR cleanup. * Add basic management tests. --------- Co-authored-by: lostluck <13907733+lostluck@users.noreply.github.com> Fix blog links (#27016) Add new Beam Python SDK examples (#26671) * Adapt cogroupbykey.py example to be compatible with Playgorund * Adapt combineglobally.py example to be compatible with Playgorund * Change cogroupbykey.py example name to avoid conflict * Fix combineglobally references in doc * Update tags for combineglobally * Adapt combineperkey.py to work with Playground * Fix test * Fix doc * Adapt combinevalues.py to work with Playground * Adapt count.py to work with Playground * Adapt distinc.py to work with Playground * Adapt groupbykey.py to work with Playground * Adapt groupintobatches.py to work with Playground * Adapt latest.py to work with Playground * Adapt max.py to work with Playground * Adapt mean.py to work with Playground * Adapt min.py to work with Playground * Adapt sample.py to work with Playground * Adapt sum.py to work with Playground * Adapt top.py to work with Playground * Adapt filter.py to work with Playground * Adapt flatmap.py to work with Playground * Adapt keys.py to work with Playground * Adapt kvswap.py to work with Playground * Adapt map.py to work with Playground * Adapt pardo.py to work with Playground * Adapt partition.py to work with Playground * Adapt regex.py to work with Playground * Adapt runinference.py to work with Playground * Adapt tostring.py to work with Playground * Adapt values.py to work with Playground * Adapt withtimestamps.py to work with Playground * Adapt groupby_test.py to work with Playground * Remove empty files * Adapt wordcount snippets to be runnable using Playground * Add example for Flatten transform * Add example for Window transform * Add example for Create transform * Add missing license * Fix example names * Fix formatting * Fix linter issues * Formattig fixes Overview of adding snippets to Playground (#25506) (#25507) * Overview of adding snippets to Playground (#25506) * Minor README changes (#25506) * Minor README changes (#25506) * List scanned directories, update example paths (#25506) * Edit after review answers (#25506) * Format comments (#25506) * Clean up (#25506) * Remove mentions of GCS support for datasets * Add documentation on how to add examples which use Kafka emulator * Add `location` field to tag example * Add license to get_your_code.md document * Revised iteration, it still is work in progress * Whitespace and other fixes * Improve Markdown formatting, fix internal links, fix typos (#25506) * Changes after internal review (#25506) * Replace the metadata table with a full YAML example (#25506) * Update the workflow diagram and the howto for adding an unlisted example (#25506) * Remove "GCS" variant from DatasetLocation as it's unsupported * Data access limitations changed * Note that context_line and categories are optional in example metadata (#25506) * Update try-beam-playground page with a link to the load_your_code doc (#25506) * Navigate to load_your_code.md on buttons in Playground (#25506) * Review and udpates * Update the diagram (#25506) * Link to ToB docs, add example URLs (#25506) * Clean up (#25506) * Fix links (#25506) * Playground doc update (#466) * Add Playground update guide to release-guide.md * Add SCHEMA.md with datastore schema description * Add information on when to implement data schema migrations and how * Add a note about indexes * Add commented out lines for local backend to backend_urls.dart * Use default version for python container * New Readme for playground * Update release-guide * Whitespace fix * Remove TODO related to Python sDK image build * Clarify TODOs * Add clarification on Redis usage * Document dependencies update process * Add script to fetch SCIO examples * Add a note on SCIO examples to load_your_code.md --------- Co-authored-by: Timur Sultanov <timur.sultanov@akvelon.com> * HOWTO for Flutter upgrade (#25506) * Clarify what to update in build.gradle for Go runner * Fix argument name * Update backend project structure * Add secription of directories used by the Playground project * Add link to manual Cloudbuild setup guide * README file update * README - HELM steps were added --------- Co-authored-by: Timur Sultanov <timur.sultanov@akvelon.com> Co-authored-by: Alex Kosolapov <alex.kosolapov@akvelon.com> Co-authored-by: Rouslan <115221004+rshamunov@users.noreply.github.com> Co-authored-by: Sergey Makarkin <Sergey.Makarkin@akvelon.com> Added dlq and error metrics to SpannerChangestreamsReadSchemaTransformProvider (#26970) * Added dlq and error metrics to SpannerChangestreamsReadSchemaTransformProvider * Add error tags in BigQuery Write and Spanner Write Transfroms Add metric counting finalized rows as reported by BQ (#26989) * Add metric counting finalized rows as reported by BQ * Run Spotless * Move stream finalization log to debug now that we have a metric to capture the information Do not skip latest blog entry from footer (#26999) Reduce the number of layers in Python base image. (#27006) Make sure spark/flink get tagged correctly (#26957) Add yaml validation using jsonschema. (#26920) * Add yaml validation using jsonschema. * Address comments. use max byte size and not max msg length in size check (#27017) Remove Beam Dependency Report code (#26954) * Remove Beam Dependency Report code * Remove reference in README * Remove other references to the dependency job * More references added initial node count variable (#27011) Co-authored-by: vdjerek <Vlado Djerek> Add minimal example for BigQuery usage (#26713) * Add minimal example for BigQuery usage * Add project name to default pipeline options * Add role for Service account - BigQuery * BigQuery DataViewer role added * Update MinimalBigQueryTornadoes.java --------- Co-authored-by: Sergey Makarkin <Sergey.Makarkin@akvelon.com> Beam Quest Blogpost (#27004) Co-authored-by: Rebecca Szper <98840847+rszper@users.noreply.github.com> add runners theory for golang (#25251) * add runners theory for golang * correct tags * Update learning/tour-of-beam/learning-content/introduction/introduction-concepts/runner-concepts/description.md Co-authored-by: Ritesh Ghorse <riteshghorse@gmail.com> --------- Co-authored-by: mende1esmende1es <mende1esmende1es@gmail.cp> Co-authored-by: Ritesh Ghorse <riteshghorse@gmail.com> Lint fixes: receiver consistency. (#27037) [Playground] Add KafkaStreaming example (#26890) * Add KafkaStreaming example Add an example for demostrating how to work with streaming data. The example produces random data, sends it into a Kafka topic and then consumes it. * Terminate example after 5 minutes * Start producing data only on next window start * Remove automatic application termination as it shuts down the whole Java VM instead of only shutting down the pipeline * Add "always_run" tag so the example will always be started using RunCode endpoint by the frontend Pass reference instead of value to cmp.Diff (#27053) Always make runner v2 default and return legacy container tag when runner_v1 is disabled (#27052) * Change the check for for runner_v2 * Add tests explicitly enabling runner v2 * change disable_runner_v2 to disable_runner_v2_until_v2.50 Disable kaniko cache compression when prebuilding images (#27023) Disable pip cache on the workers to reduce prebuilt container image size (#27035) Co-authored-by: tvalentyn <tvalentyn@users.noreply.github.com> Beam quest Blog Addon (#27051) Co-authored-by: Rebecca Szper <98840847+rszper@users.noreply.github.com> Co-authored-by: Ahmet Altay <aaltay@gmail.com> Make use of prism in periodic tests not cross compile. (#27054) Co-authored-by: lostluck <13907733+lostluck@users.noreply.github.com> Move AdaptiveThrottler impl to new Components directory (#27018) * Move AdaptiveThrottler impl to new Components directory * Remove unused consts from util.py * Linting * More linting Merge pull request #27047: Enable pubsub dynamic destinations by default [Playground] Make new python examples runnable (#27028) * fix examples not being invoked when running in Playground * Fix wordcount_wordcount example options * Make wordcount_debugging work exactly as it does in tests * Formatting fix * Add output to groupby_simple_aggregate * Fix formatting * Write text to output.txt reads all files from metadata list (#27059) [Tour of Beam] Learning content for "Splittable DoFn" module (#25707) * correct examples * delete whitespace * correct space * remove line and correct one line * correct context line --------- Co-authored-by: mende1esmende1es <mende1esmende1es@gmail.cp> Unpin tox version in Beam code base (#26082) * Update BeamModulePlugin.groovy * add run to the tox cmd * Pin tox upper bound * Revert "Pin tox upper bound" This reverts commit 15020171aa6d0224f3a4ddd16106c701c9897c99. * Add /bin/bash to allow list externals * Remove tox versioning * Add comma * Add curl to the allowlist externals * Update sdks/python/tox.ini * Update sdks/python/tox.ini Issue26858 tob tests examples (#26932) * temp comment * handled examples failed to load error * issue26858 minor changes * Fix reporting examples failed to load in ToB frontend (#26858) * move ignore for file * returned commented tests * Fix playground_components tests (#26858) * Fix loading ExampleViewOptions for standard examples (#26858) * fix tests * try to fix tests * some prints for debugging added * removed prints * retrigger checks --------- Co-authored-by: Alexey Inkin <leha@inkin.ru> Co-authored-by: darkhan.nausharipov <darkhan.nausharipov@kzn.akvelon.com> Co-authored-by: Alexey Inkin <alexey.inkin@akvelon.com> Fix direct SDF runner handle_process_outputs (#27062) * Fix Direct Runner SDF output handler * Remove never worked test Remove leftovers * Fix lint [#27064] Make Tag existence explicit in release docs. (#27070) * [#27064] Make Tag existence explicit in release docs. * Add tag check. * ws --------- Co-authored-by: lostluck <13907733+lostluck@users.noreply.github.com> Add Certification + Beam Quest to Documentation (#26997) Co-authored-by: Rebecca Szper <98840847+rszper@users.noreply.github.com> Add note for updated release tag for 2.48 (#27073) Quote Addition + Sharing instructions [Beam Quest Blog] (#27066) Update Playground examples (#25032) * Merge changes * Remove incorrect tags * transforms tags unified * Tags amended * Fix context_lines for all examples * Update Kafka examples * Fixes for Kafka examples * Java spotless * Fix Java compilation issues * Fix Java compilation issues * Add Java dependencies Update TensorRT dockerfile (#27072) Option to return timezone aware datetime for to_utc_datetime (#26962) * Option to return timezone aware datetime for to_utc_datetime * Fix lint * Add pydoc reminding possible timezone mismatch * Fix pydoc Update to libraries-bom 26.16.0 (#27074) [Tour of Beam] Learning content for "Cross-language Transforms" module (#26227) Bump google.golang.org/api from 0.125.0 to 0.126.0 in /sdks (#27081) Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.125.0 to 0.126.0. - [Release notes](https://github.com/googleapis/google-api-go-client/releases) - [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md) - [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.125.0...v0.126.0) --- updated-dependencies: - dependency-name: google.golang.org/api dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Bigquery schema generation fix (#26960) * Detect collisions in Avro schema generation and use fully-qualified schema names when that happens. This should not interfere with existing behavior as this code path cannot currently encode schemas with the same name for a RECORD field in the schema (position and depth do not matter as a flat namespace is used) Github Workflow Replacement for Jenkins Job, beam_PreCommit_Go (Cron, Commit, Phrase) (#27042) * triggers fix for comment, cron, push * cron condition fix * cron trigger fix Preserve output when a pipeline dies on timeout (#26928) (#26929) [Playground] [Frontend] FCE updated to v0.3.0, minor improvements (#27063) * FCE updated to v0.3.0 * no wildcard_variable_uses * Add ToB HTML title --------- Co-authored-by: darkhan.nausharipov <darkhan.nausharipov@kzn.akvelon.com> Co-authored-by: Alexey Inkin <alexey.inkin@akvelon.com> Embed examples (#26933) * Embed the word count examples * Fix links * Add Go examples to the WordCount tutorial * Embed the examples to the transform catalog * Drop the non-runnable Python examples from the WordCount tutorial, embed the Java ones by path * Minor embedding fixes * Delete the old example in CoGroupByKey * Fix minor embedding issues * Do not autofocus to embedded playground * Undo old example deletion * Clean up * Clean up * Fix embedded playground failure in WebKit * Fix not starting tostring_iterables.py * Undo an old example deletion * Fix SDK-dependent titles --------- Co-authored-by: Timur Sultanov <timur.sultanov@akvelon.com> [Playground] - New role add (#27082) * New role add * add comma --------- Co-authored-by: Sergey Makarkin <sergey.makarkin@akvelon.com> Release Guide update for Playground environment (#27043) * Added steps to Release management doc * Added CD step Update learning-resources.md with links to video material (#27079) Cut the Playground output if it is too long (#27083) (#27084) * Cut the Playground if it is too long (#27083) * Improve error logging (#27083) Refactor key coder for Samza portable runner (#27095) Add required commands to allowlist_externals in tox.ini (#27089) * Remove multiple allowlist and place them at a single place * Add additional commands to allowlist Adding error tags in BigQuery Write Transforms (#27020) * Added dlq and error metrics to SpannerChangestreamsReadSchemaTransformProvider * Add error tags in BigQuery Write and Spanner Write Transfroms * Adding error tag in BigQueryStorageWriteApiSchemaTransformProvider Update BQ Autosahrding notes (#26874) Add fallback reviewers. (#26835) Bump google.golang.org/api from 0.123.0 to 0.124.0 in /sdks (#26856) Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.123.0 to 0.124.0. - [Release notes](https://github.com/googleapis/google-api-go-client/releases) - [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md) - [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.123.0...v0.124.0) --- updated-dependencies: - dependency-name: google.golang.org/api dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Naireen Hussain <naireenhussain@google.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Add notebook for run inference with streaming pipeline example (#27075) * add notebook for run inference with streaming pipeline example * updated name, doc comments, removed pubsub * changed title, filename * Update examples/notebooks/beam-ml/run_inference_generative_AI.ipynb Co-authored-by: Rebecca Szper <98840847+rszper@users.noreply.github.com> --------- Co-authored-by: Rebecca Szper <98840847+rszper@users.noreply.github.com> [Python] Rename example notebook to fix links (#27102) Fix graph, update the Playground frontend README (#27100) * Update README * Fix typos * Fix graph * Trigger tests [Playground] Remove old deployment workflows (#27027) * [Playground] Remove old deployment workflows These workflows had been replaced by new CloudBuild triggers based ones. * Remove doc for non-existing workflows Bump github.com/tetratelabs/wazero from 1.1.0 to 1.2.0 in /sdks (#27010) Bumps [github.com/tetratelabs/wazero](https://github.com/tetratelabs/wazero) from 1.1.0 to 1.2.0. - [Release notes](https://github.com/tetratelabs/wazero/releases) - [Commits](https://github.com/tetratelabs/wazero/compare/v1.1.0...v1.2.0) --- updated-dependencies: - dependency-name: github.com/tetratelabs/wazero dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Bump go.mongodb.org/mongo-driver from 1.11.6 to 1.11.7 in /sdks (#27041) Bumps [go.mongodb.org/mongo-driver](https://github.com/mongodb/mongo-go-driver) from 1.11.6 to 1.11.7. - [Release notes](https://github.com/mongodb/mongo-go-driver/releases) - [Commits](https://github.com/mongodb/mongo-go-driver/compare/v1.11.6...v1.11.7) --- updated-dependencies: - dependency-name: go.mongodb.org/mongo-driver dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Bump cloud.google.com/go/bigquery from 1.51.1 to 1.51.2 in /sdks (#26627) Bumps [cloud.google.com/go/bigquery](https://github.com/googleapis/google-cloud-go) from 1.51.1 to 1.51.2. - [Release notes](https://github.com/googleapis/google-cloud-go/releases) - [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md) - [Commits](https://github.com/googleapis/google-cloud-go/compare/bigquery/v1.51.1...bigquery/v1.51.2) --- updated-dependencies: - dependency-name: cloud.google.com/go/bigquery dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> added beam_PreCommit_Typescript Github Actions job added beam_PreCommit_Typescript Github Actions job added beam_PreCommit_Typescript Github Actions job added beam_PreCommit_Typescript Github Actions job test test test test test test test test test test test added beam_PreCommit_Java_Examples_Dataflow job added beam_PreCommit_Java_Examples_Dataflow job added beam_PreCommit_Java_Examples_Dataflow job added beam_PreCommit_Java_Examples_Dataflow job added beam_PreCommit_Java_Examples_Dataflow job added beam_PreCommit_Java_Examples_Dataflow job added beam_PreCommit_Java_Examples_Dataflow job added beam_PreCommit_Java_Examples_Dataflow job added beam_PreCommit_Java_Examples_Dataflow job added beam_PreCommit_Java_Examples_Dataflow job added beam_PreCommit_Java_Examples_Dataflow job added beam_PreCommit_Java_Examples_Dataflow job added beam_PreCommit_Java_Examples_Dataflow job added beam_PreCommit_Java_Examples_Dataflow job added beam_PreCommit_Java_Examples_Dataflow job added beam_PreCommit_Java_Examples_Dataflow job added beam_PreCommit_Java_Examples_Dataflow job added beam_PreCommit_Java_Examples_Dataflow job added beam_PreCommit_Java_Examples_Dataflow job added beam_PreCommit_Java_Examples_Dataflow job added beam_PreCommit_Java_Examples_Dataflow job added beam_PreCommit_Java_Examples_Dataflow job added beam_PreCommit_Java_Examples_Dataflow job added beam_PreCommit_Java_Examples_Dataflow job added beam_PreCommit_Java_Examples_Dataflow job added beam_PreCommit_Java_Examples_Dataflow job added beam_PreCommit_Java_Examples_Dataflow job added beam_PreCommit_Java_Examples_Dataflow job added beam_PreCommit_Java_Examples_Dataflow job added beam_PreCommit_Java_Examples_Dataflow job added beam_PreCommit_Java_Examples_Dataflow job added beam_PreCommit_Java_Examples_Dataflow job added beam_PreCommit_Java_Examples_Dataflow job test beam_PreCommit_Java_Examples_Dataflow added required steps Bump google.golang.org/api from 0.126.0 to 0.127.0 in /sdks (#27112) Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.126.0 to 0.127.0. - [Release notes](https://github.com/googleapis/google-api-go-client/releases) - [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md) - [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.126.0...v0.127.0) --- updated-dependencies: - dependency-name: google.golang.org/api dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Merge pull request #26975: Add upsert and delete support to BigQueryIO Merge pull request #26849: add attribute support to writeAvros and writeProtos Bump golang.org/x/oauth2 from 0.8.0 to 0.9.0 in /sdks (#27124) Bumps [golang.org/x/oauth2](https://github.com/golang/oauth2) from 0.8.0 to 0.9.0. - [Commits](https://github.com/golang/oauth2/compare/v0.8.0...v0.9.0) --- updated-dependencies: - dependency-name: golang.org/x/oauth2 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Bump cryptography from 40.0.2 to 41.0.0 in /sdks/python/container/py37 (#26995) Bumps [cryptography](https://github.com/pyca/cryptography) from 40.0.2 to 41.0.0. - [Changelog](https://github.com/pyca/cryptography/blob/main/CHANGELOG.rst) - [Commits](https://github.com/pyca/cryptography/compare/40.0.2...41.0.0) --- updated-dependencies: - dependency-name: cryptography dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Portable bundle manager in Samza Runner (#26475) Add OWNERS file for Samza Runner (#27130) Initial implementation of QuotaEvent (#27078) * Initial implementation of QuotaEvent * Fix style * Fix typo * Add benchmark * Added clarification about close() QuotaEvents for BigQuery Upgrade Node dependency to 14.x (#26988) Bump actions/stale from 6 to 8 (#27111) Bumps [actions/stale](https://github.com/actions/stale) from 6 to 8. - [Release notes](https://github.com/actions/stale/releases) - [Changelog](https://github.com/actions/stale/blob/main/CHANGELOG.md) - [Commits](https://github.com/actions/stale/compare/v6...v8) --- updated-dependencies: - dependency-name: actions/stale dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Github Workflow Replacement for Jenkins Job, beam_PreCommit_Whitespace (Cron, Commit, Phrase) (#27099) * added workflow beam_PreCommit_Whitespace * apache license block add * push path add Bump transformers (#27109) Bumps [transformers](https://github.com/huggingface/transformers) from 4.24.0 to 4.30.0. - [Release notes](https://github.com/huggingface/transformers/releases) - [Commits](https://github.com/huggingface/transformers/compare/v4.24.0...v4.30.0) --- updated-dependencies: - dependency-name: transformers dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Bump transformers (#27107) Bumps [transformers](https://github.com/huggingface/transformers) from 4.21.1 to 4.30.0. - [Release notes](https://github.com/huggingface/transformers/releases) - [Commits](https://github.com/huggingface/transformers/compare/v4.21.1...v4.30.0) --- updated-dependencies: - dependency-name: transformers dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Github Workflow Replacement for Jenkins Job, beam_PreCommit_Website (Cron, Commit, Phrase) (#27098) * workflow added beam_PreCommit_Website * apache license block add * push path website add Implement Java exception sampling (#27121) * Implement Java exception sampling * spotless * s/Exception/IOException Implement basic mapping capabilities for YAML. (#27096) Bump golang.org/x/sync from 0.2.0 to 0.3.0 in /sdks (#27137) Bumps [golang.org/x/sync](https://github.com/golang/sync) from 0.2.0 to 0.3.0. - [Commits](https://github.com/golang/sync/compare/v0.2.0...v0.3.0) --- updated-dependencies: - dependency-name: golang.org/x/sync dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Move to a conditionVariable for messages+state stream + test. (#27060) Co-authored-by: lostluck <13907733+lostluck@users.noreply.github.com> fix: fix beam metrics after migrating to async batcher (#27085) * fix: fix beam metrics after migrating to async batcher * update version * update bigtable version * update format * fix format Bump github.com/aws/aws-sdk-go-v2/service/s3 in /sdks (#27149) Bumps [github.com/aws/aws-sdk-go-v2/service/s3](https://github.com/aws/aws-sdk-go-v2) from 1.33.1 to 1.34.1. - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases) - [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/CHANGELOG.md) - [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/s3/v1.33.1...service/s3/v1.34.1) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go-v2/service/s3 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Adding error tag and metrics in SpannerWriteSchemaTransformProvider (#27021) .test-infra Pipeline reads from Dataflow API and writes to BigQuery (#27008) * Create terraform to setup eventarc pipeline * Initial commit of pipeline * Fix stackoverflow * wip * aip * wip * wip * wip * wip * code cleanup to pass check * fix get metrics * Fix schema of embedded types * pass checks * Run locally, data written to BigQuery * Make Descriptor Schema registry static final instance * Configure flex template * Configure local template build * Allow dataflow worker to read artifact registry * Give dataflow worker access to api * Modify storage bucket * Throttle requests * Change Executors to single threaded * Additional cleanup * Gracefully shutdown threadpool * Add requests success/failure counters * Add items metrics * Cache and centralize channel shutdowns * Cleanup terraform * Cleanup code comments * Fix rat/whitespace checks * Remove unnecessary added dependency * Add @Internal to all classes Pylint repression same line install release candidates if available for setupVirtualEnv and instalLGCPTest tasks (#27068) Fix Python37 PostCommit azure-cli not support Bookworm (#27153) * Fix tox whitelist breaking change * Disable azurite azure API version check Fix sklearn requirement file in Py311 (#27152) * Fix sklearn requirement file in Py311 * Skip the test where model incompatible with newer sklearn version Select dedicated avro datum reader and writer (#26320) * Select dedicated avro darum reader and writer Use proper datum reader and writer and exose factories so users can provide custome ones. * Drop useReflectApi * Make test datum factory serializable * Reduce breaking changes * Keep old PubSubIO API * Do not deprecate API so far * Revert pubsub test * Remove leftover deprecated annotaions * Checkstyle imports * Remove abstract modifier to AvroCoder * Remove AvroCoder sub-types * Delegate coder type info to datum factory * Normalize coder factory usage * spotless & checkstyle * Fix compilation issue with parametrized type * Fix could object test for specialized AvroGenericCoder * Fix comment * Harden avroVersionsTest and add custom factory example * Revert AvroCoder constructor behavior change * Leverage AvroDatumFactory if passed are reader factory in AvroSource * Apply checkstyle * Cleanup build script * Add entry in CHANGES.md * Add extra test for nullable logical-type conversion in 1.9+ Update numpy requirement in /sdks/python (#27160) Updates the requirements on [numpy](https://github.com/numpy/numpy) to permit the latest version. - [Release notes](https://github.com/numpy/numpy/releases) - [Changelog](https://github.com/numpy/numpy/blob/main/doc/RELEASE_WALKTHROUGH.rst) - [Commits](https://github.com/numpy/numpy/compare/v1.14.3...v1.25.0) --- updated-dependencies: - dependency-name: numpy dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Bump google.golang.org/api from 0.127.0 to 0.128.0 in /sdks (#27163) Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.127.0 to 0.128.0. - [Release notes](https://github.com/googleapis/google-api-go-client/releases) - [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md) - [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.127.0...v0.128.0) --- updated-dependencies: - dependency-name: google.golang.org/api dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Bump github.com/tetratelabs/wazero from 1.2.0 to 1.2.1 in /sdks (#27162) Bumps [github.com/tetratelabs/wazero](https://github.com/tetratelabs/wazero) from 1.2.0 to 1.2.1. - [Release notes](https://github.com/tetratelabs/wazero/releases) - [Commits](https://github.com/tetratelabs/wazero/compare/v1.2.0...v1.2.1) --- updated-dependencies: - dependency-name: github.com/tetratelabs/wazero dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Bump actions/setup-go from 3 to 4 (#27110) Bumps [actions/setup-go](https://github.com/actions/setup-go) from 3 to 4. - [Release notes](https://github.com/actions/setup-go/releases) - [Commits](https://github.com/actions/setup-go/compare/v3...v4) --- updated-dependencies: - dependency-name: actions/setup-go dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> [website] #27115 Add supported Flink versions (#27117) Bump github.com/aws/aws-sdk-go-v2/service/s3 in /sdks (#27172) Bumps [github.com/aws/aws-sdk-go-v2/service/s3](https://github.com/aws/aws-sdk-go-v2) from 1.34.1 to 1.35.0. - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases) - [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/CHANGELOG.md) - [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/s3/v1.34.1...service/s3/v1.35.0) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go-v2/service/s3 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Avro datum factory fixup (#27173) * Remove DecimalConversion from logical-type conversions * Restore AvroCloudObject compatibility Add initial DLQ router (#27045) * Add initial DLQ router * fix apache licenses Fix locking race between duplicate PartitionRecords w matching uid (#27055) Change default work remaining to 0.0 instead of 1.0 (#27032) * Factor out the SubscriptionPartitionRestrictionTracker and add a hasProgress implementation * Change default workremaining to 0 instead of 1 for improving downscaling * fix nullable annotation * Improve DoFnInvoker comments on the change from a default backlog of 1.0 to 0.0 * apply spotless [Code Completion Plugin] Define Element Pattern for java sdk obsolete action removed (#27143) Issue26977 changing sdk bug (#26978) * temp comment * handled examples failed to load error * issue26858 minor changes * issue26938 fixed examples loading twice * issue26977 fix change sdk bug * revert testing feature * remove unused imports * rename method * issue26977 tests added * fixed test * issue26977 fixed sdk changing tests * method extracted * tests simplified * fixed tests * Upgrade Flutter to v3.10.4 (#26977) --------- Co-authored-by: Alexey Inkin <alexey.inkin@akvelon.com> [Website] add project shield case study (#27087) Ensure that the BeamFnLoggingClient terminates process if stream breaks (#25186) * Ensure that a failure of the logging stream in BeamFnLoggingClient does not cause logging to block indefinitely but instead triggers SDK teardown. * fix test * fix racy test * address comments Bump cloud.google.com/go/spanner from 1.46.0 to 1.47.0 in /sdks (#27198) Bumps [cloud.google.com/go/spanner](https://github.com/googleapis/google-cloud-go) from 1.46.0 to 1.47.0. - [Release notes](https://github.com/googleapis/google-cloud-go/releases) - [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md) - [Commits](https://github.com/googleapis/google-cloud-go/compare/spanner/v1.46.0...spanner/v1.47.0) --- updated-dependencies: - dependency-name: cloud.google.com/go/spanner dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> [prism] UI improvements. (#27144) exception handling for loading models (#27186) [Tour of Beam] [Frontend] UI refinement from 01.06.23 (#26964) * default sdk, solution warning, bottom panel alignment * deleted code * markdown font sizes * review * removed Until we select an SDK the dropdown is not shown * _checkSdksLoadedCorrectly fix * _sdk in tour page test * relaunch ci * relaunch ci * deleted import, print exception * separate google forms url for tob * relaunch ci * relaunch ci * prints in tabbed_snippet_editor for debugging * reverted TabbedSnippetEditor with todo --------- Co-authored-by: darkhan.nausharipov <darkhan.nausharipov@kzn.akvelon.com> Add banner and links to Tour of Beam to website (#26113) (#26145) * Add banner and links to Tour of Beam to website (#26113) * Lighten up the Tour of Beam desktop banner background (#26113) * Revert "Lighten up the Tour of Beam desktop banner background (#26113)" * Responsive height for top banners on desktop (#26113 * Semi-transparent pagination for banners (#26113) disable examples check (#27126) * disable examples check * Clean up (#27125) * Fix a Gradle task (#27125) * Fix a SCIO example descriptor (#27125) * Test local code instead of fetching from GitHub (#27125) * Fix tests (#27125) * Fix tests (#27125) * Update README (#27125) * Re-run tests (#27125) --------- Co-authored-by: Alexey Inkin <alexey.inkin@akvelon.com> Fix incorrect metadata for ToB Splittable DoFN module (#27092) * add work example * correct * fixing incorrect module structure * fixed io example tag * correct tags for bigquery examples --------- Co-authored-by: mende1esmende1es <mende1esmende1es@gmail.cp> Performance improvements (#27119) * Performance improvements * Add a missing file Fix not sharing code from a duplicate tab (#27127) (#27128) [runners-flink] #27175 fix coder for eagerly bound state (#27176) * [runners-flink] #27175 fix coder for eagerly bound state * [website] update pipeline options for FlinkRunner [Security] Upgrade snappy-java to 1.1.10.0 (#27030) Revert "Adding error tag and metrics in SpannerWriteSchemaTransformProvider (#27021)" (#27204) This reverts commit 7ecd8e514448406a15bedc85a680b1e7153774f9. Add PortableUtils for Samza Runner (#27131) Bump github.com/aws/aws-sdk-go-v2/config in /sdks (#27148) Bumps [github.com/aws/aws-sdk-go-v2/config](https://github.com/aws/aws-sdk-go-v2) from 1.18.25 to 1.18.27. - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases) - [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/CHANGELOG.md) - [Commits](https://github.com/aws/aws-sdk-go-v2/compare/config/v1.18.25...config/v1.18.27) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go-v2/config dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Bigtable Read Xlang Wrapper for Python SDK (#26593) * create a more friendly BigtableRow * allow ByteBuffer Map value type in Beam Row * ReadFromBigtable and tests * skip if expansion port doesn't exist * formatting fixes * formatting * move attribute up * validate invalid configs * format fix * write 20 rows for test * format fix * debug find sdk location * debug: dont use wheels distribution * add sdk_container_image to df tests * use beamjarexpansionservice directly * ensure runs on python virtual env * format fix * remove unused labels field * add test for _BeamRowToPartialRowData * skip tests if bt dependencies not installed * test row key equality * fix test [Playground] Decrease severity of not important log messages (#27116) * Decrease log message severity for catalog cache timeout messages * Decrease severity of readiness check messages * Log transaction errors as warnings * Decrease severity of run error messages as they litter logs when user supplied code crashes * Revert "Decrease severity of readiness check messages" This reverts commit 9663bbad8a24e8dd1ee6cb634c23f05a4c420c03. [Playground] - Cloudfunction rename (#27139) * Change CloudFunction name * Fix function name * revert gradle file and change terraform file * Minor changes in Readme file * readme file update --------- Co-authored-by: Sergey Makarkin <sergey.makarkin@akvelon.com> Adding error tag and metrics in SpannerWriteSchemaTransformProvider (#27184) Add (GPN:Beam) to the user-agent string of Python GCS SDKs. (#27211) * Add (GPN:Beam) to the user-agent string of Python GCS SDKs. For a valid user-agent string, the version number should be separated with the product name by "/". In addition, the requested string of "(GPN:Beam)" is considered a comment, and could not be the start of the user-agent string. * Add (GPN:Beam) to the user-agent string of Python GCS SDKs. For a valid user-agent string, the version number should be separated with the product name by "/". In addition, the requested string of "(GPN:Beam)" is considered a comment, and could not be the start of the user-agent string. Bump google.golang.org/grpc from 1.55.0 to 1.56.1 in /sdks (#27216) Bumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.55.0 to 1.56.1. - [Release notes](https://github.com/grpc/grpc-go/releases) - [Commits](https://github.com/grpc/grpc-go/compare/v1.55.0...v1.56.1) --- updated-dependencies: - dependency-name: google.golang.org/grpc dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Fixes naming conflict with "Tour of Beam" (#27206) * rename * rename * rename * rename * renaming + link to the tour of beam. * renaming + link to the tour of beam. * move quotes * minor fixes * delete old tour of beam page * delete from nav page Remove python 3.7 support (#27194) * Remove python 3.7 support * Fix few places where py37 was removed * Add notes to changes.MD * fix pylint mypy * Add lint task to py38 * Fix error: The u prefix for strings is no longer necessary in Python >=3.0 * Fix few more lint errors * Don't modify bq client files disabling triggers on precommit website due to regression (#27185) Refactor code to make the UI more readable. Look at https://github.com/apache/beam/issues/27005#issuecomment-1599358169 (#27192) Add an affinity concept to yaml providers. (#27105) This will allow for selection of more-likely-to-fuse implementations for adjacent operations when more than one provider services the same operation. Update to libraries-bom 26.17.0 (#27209) Bump github.com/testcontainers/testcontainers-go in /sdks (#27229) Bumps [github.com/testcontainers/testcontainers-go](https://github.com/testcontainers/testcontainers-go) from 0.20.1 to 0.21.0. - [Release notes](https://github.com/testcontainers/testcontainers-go/releases) - [Commits](https://github.com/testcontainers/testcontainers-go/compare/v0.20.1...v0.21.0) --- updated-dependencies: - dependency-name: github.com/testcontainers/testcontainers-go dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Bump github.com/aws/aws-sdk-go-v2/feature/s3/manager in /sdks (#27161) Bumps [github.com/aws/aws-sdk-go-v2/feature/s3/manager](https://github.com/aws/aws-sdk-go-v2) from 1.11.67 to 1.11.70. - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases) - [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/CHANGELOG.md) - [Commits](https://github.com/aws/aws-sdk-go-v2/compare/feature/s3/manager/v1.11.67...feature/s3/manager/v1.11.70) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go-v2/feature/s3/manager dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Using capabilities instead of the container name to set use_single_core_per_container Add Exception metadata to the data sampling protocol Add https://github.com/apache/beam/issues/27000 as a known issue released in Beam 2.48.0 (#27235) Python data sampling optimization (#27157) * Python optimization work * Exception Sampling perf tests * add better element sampling microbenchmark * slowly move towards plumbing the samplers to the bundle processors * cleaned up * starting clean up and more testing * finish tests * fix unused data_sampler args and comments * yapf, comments, and simplifications * linter * lint and mypy * linter * run tests * address review comments * run tests --------- Co-authored-by: Sam Rohde <srohde@google.com> Improve autoscaler throughput estimates and account for heartbeats (#27056) The throughput estimates are improved by storing the estimates per partition (in StreamProgress) rather than per dofn instance. This required some refactoring of ThroughputEstimator/SizeEstimator. It now estimates throughput (per second) of the last ReadChangeStream request. fix(typescript): Publish types correctly, don't publish tests fix(typescript): added more exports Basic error handling for yaml. (#27145) Updates Python ExternalTransform to use the transform service when needed (#27228) * Updates Python ExternalTransform to use the transform service when needed * Addressing reviewer comments * Fix yapf * Fix lint * Fix yapf Add blog post for Managing Beam dependencies in Java Vertex AI Remote Model Handler (#27091) * Move WIP code to new branch at HEAD * Add client-side throttling * Various linting issues * More linting, add dependency to setup.py * Fix Docstring * new flags, types, framework for eventual file read * Route to correct tutorial * Better flag name * Align type hints, pylint cleanup * Move off of constant, preprocess image * Change to keyed version for output * Yapf * Pylint * Clean up ToDo * Add TODOs for file globs * Fix cast, remove unneccessary assert * whitespace * More whitespace * Import order * More import ordering * Amend comment [Playground] [Frontend] Brand colors in Playground flutter_code_editor (#27218) * brand colors in PG FCE * link to website * restart ci * restart ci --------- Co-authored-by: darkhan.nausharipov <darkhan.nausharipov@kzn.akvelon.com> Bump go.mongodb.org/mongo-driver from 1.11.7 to 1.12.0 in /sdks (#27215) Bumps [go.mongodb.org/mongo-driver](https://github.com/mongodb/mongo-go-driver) from 1.11.7 to 1.12.0. - [Release notes](https://github.com/mongodb/mongo-go-driver/releases) - [Commits](https://github.com/mongodb/mongo-go-driver/compare/v1.11.7...v1.12.0) --- updated-dependencies: - dependency-name: go.mongodb.org/mongo-driver dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> update readme with Beam Playground, Tour of Beam (#27243) Co-authored-by: xqhu <xqhu@google.com> runner image and arc changes (#27118) * runner image and arc changes * add state bucket comment in env file for reference * add readme for manual image push and sync beam environment --------- Co-authored-by: vdjerek <Vlado Djerek> Fix whitespace (#27247) test beam_PreCommit_Website test beam_PreCommit_Website test beam_PreCommit_Website [Tour of Beam] Learning content for "Final challenge" module (#26861) * add final challenge * correct examples * changge * correct golang * correct example hint * correct examples and add golang example * delete whitespace * delete whitespace * add file tag * correct final challenge * fixing incorrect tags and names * minor formatting * fixing example formatting * correct challenge * correct whitespace * change final-challenge-2 * correct imports * fix splittable unit id * formatting * format change * formatting * fixing template * temp fix of url * backend urls fix * restore backend urls * change * remove window --------- Co-authored-by: mende1esmende1es <mende1esmende1es@gmail.cp> Co-authored-by: Oleh Borysevych <oleg.borisevich@akvelon.com> Bump transformers (#27108) Bumps [transformers](https://github.com/huggingface/transformers) from 4.21.1 to 4.30.0. - [Release notes](https://github.com/huggingface/transformers/releases) - [Commits](https://github.com/huggingface/transformers/compare/v4.21.1...v4.30.0) --- updated-dependencies: - dependency-name: transformers dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Code completion plugin: add plugin tests workflow (#27212) * Code completion plugin: add plugin tests workflow * Testing changing directory in code_completion_plugin_tests.yml * Making gradlew executable * Fix gradlew intellij plugin import * change gradlew premissions * code-completion-plugin: fixing GA tests workflow * Revert "change gradlew premissions" This reverts commit ee396be64ec3bccb883d687fc0c201c6c0d35562. * Revert "code-completion-plugin: fixing GA tests workflow" This reverts commit 9baa351a003d30980ff0bcbe582a80c5853e0fd0. * fix code_completion_plugin_tests.yml * Making gradlew executable * code-completion-plugin-fix intellij plugin version in build.gradle.kts * code-completion-plugin: change java version (17 -> 11) in code_completion_plugin_tests.yml * test code completion plugin CI * Add env vars to code_completion_plugin_test.yml * code-completion-plugin: test ci * code-completion-plugin: clone intellij-community to CI runner --------- Co-authored-by: Pablo <pabloem@users.noreply.github.com> Co-authored-by: Pablo E <pabloem@apache.org> add job rerun action (#27210) * add job rerun action * Update .github/actions/rerun-job-action/action.yml Typo Co-authored-by: Danny McCormick <dannymccormick@google.com> * header and a comment explaining the logic for going into re-run flow * deduplicate API calls * added some explanation within the action file * Update .github/actions/rerun-job-action/action.yml Typo Co-authored-by: Danny McCormick <dannymccormick@google.com> --------- Co-authored-by: vdjerek <Vlado Djerek> Co-authored-by: Danny McCormick <dannymccormick@google.com> Code completion plugin: Add Java SDK Transform Completions (Hard Coded) (#27168) * [Code Completion Plugin] Define Element Pattern for java sdk * Code Completion Plugin: Add java sdk transform completions * Verifying that tests are running as expected :) Sorry! * Update BeamCompletionContributorTestCase.java --------- Co-authored-by: Pablo <pabloem@users.noreply.github.com> [Website] add hsbc case study (#27200) [Tour of Beam] add work example (#27080) * add work example * correct * correct tags for bigquery examples * correct read-query * correct read-query tag * correct imports * remove package * correct * fixed example name --------- Co-authored-by: mende1esmende1es <mende1esmende1es@gmail.cp> Co-authored-by: Oleh Borysevych <oleg.borisevich@akvelon.com> Remove py37 references (#27252) Handling issue where keystore file is empty for MongoDB SSL (#27250) Bump worker image dependency (#27253) [#22737] Add line about timer support for Go (#27263) Bump actions/checkout from 2 to 3 (#27259) Bumps [actions/checkout](https://github.com/actions/checkout) from 2 to 3. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](https://github.com/actions/checkout/compare/v2...v3) --- updated-dependencies: - dependency-name: actions/checkout dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Add Tour of Beam Page (#27244) * add tour of beam page to redirect. * add tour of beam page to redirect. * add tour of beam page to redirect. Bump transformers (#27106) Bumps [transformers](https://github.com/huggingface/transformers) from 4.21.0 to 4.30.0. - [Release notes](https://github.com/huggingface/transformers/releases) - [Commits](https://github.com/huggingface/transformers/compare/v4.21.0...v4.30.0) --- updated-dependencies: - dependency-name: transformers dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Bump google.golang.org/protobuf from 1.30.0 to 1.31.0 in /sdks (#27262) Bumps google.golang.org/protobuf from 1.30.0 to 1.31.0. --- updated-dependencies: - dependency-name: google.golang.org/protobuf dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> [prism] Fix linting issue for flag var (#27248) updated job conditions test test test test improved if conditions improved if conditions test test test test Add notebook for image processing using beam (#27034) * Add notebook for image processing using beam * Delete image_processing_tensorflow.ipynb * add image processing use case using tensorflow * Delete image_processing_tensorflow.ipynb * Add image processing use case after suggestions * Delete image_processing_beam.ipynb * Add image processing with implemented suggestions Enable remote caching for Beam builds (#27015) Remove some null checks for optional SplunkIO parameters (#27177) don't error-log expected offset mismatch errors (#27251) Update Release Guide DockerHub beammaintainers team to beam (#27264) Add error metrics and DLQ support to GCS/File write schema transforms (#27039) * init * fixing * fixed avro schema transform * json error metrics * parquet error metric * xml error metrics * fix comments * spotless * add tests * spotless * Change name of ErrorCounterFn * add generic test for BeamRowMapperWithDlq Disable FirestoreIT tests on Dataflow (#27267) * Disable FirestoreIT tests on Dataflow * Fix spotless Fix perf tool tests (#27270) * Fix tests * Don't publish data when running analyzer in PR * Update .github/workflows/run_perf_alert_tool.yml * Run perf analysis tests during schedules * Don't authenticate during pull_request Bump google.golang.org/api from 0.128.0 to 0.129.0 in /sdks (#27265) Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.128.0 to 0.129.0. - [Release notes](https://github.com/googleapis/google-api-go-client/releases) - [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md) - [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.128.0...v0.129.0) --- updated-dependencies: - dependency-name: google.golang.org/api dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Use a more mnemonic variable name (#27217) Co-authored-by: Julian Berzborn <julian.berzborn@vodafone.com> Bump cloud.google.com/go/bigquery from 1.51.2 to 1.52.0 in /sdks (#27266) Bumps [cloud.google.com/go/bigquery](https://github.com/googleapis/google-cloud-go) from 1.51.2 to 1.52.0. - [Release notes](https://github.com/googleapis/google-cloud-go/releases) - [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md) - [Commits](https://github.com/googleapis/google-cloud-go/compare/bigquery/v1.51.2...bigquery/v1.52.0) --- updated-dependencies: - dependency-name: cloud.google.com/go/bigquery dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Start expansion service containers before the controller container Cleanup for GA launch of ReadChangeStream (#27249) This includes a series of small changes to release the connector: - Remove option for users to override heartbeat duration - Add option to skip creating metadata table and standalone utility to create metadata table - Add hard timeout to mutateRow requests as workaround for hanging requests - Remove excessive RCSP logging for debugging during preview - Increase read rows timeout to better accomodate large tables - Remove side effects from InitializeDoFn - Lower ReadChangeStream deadline to closer align with checkpoint duration - Add release note to CHANGES.md Decrease run log failures severity (#27230) Decrease severity of messages which come from child processes as they usually indicate error in user-submitted code BigTable write SchemaTransform and Python wrapper (#27146) * python wrapper * schematransform; started some tests * schematransform and tests done * python wrapper and tests done * add tests for _DirectRowMutationsToBeamRow DoFn * use expansion service * style fix * style fix * use secrets token in instance name to prevent race condition between py37 & py311 tests * use typing typehints * style fix * add missing import * test fix * resolve merge issues * style fix * raise timeout of dataflow tests * fix setting timeout * address reviewer comments * address reviewer comments * trivial change * remove unused imports; lint fixes * add unittest main call [Security] Upgrade guava to 32.0.1-jre (#27279) * [Security] Upgrade guava to 32.0.1-jre * Update j2objc-annotations reference to match Guava's Bump cloud.google.com/go/pubsub from 1.31.0 to 1.32.0 in /sdks (#27282) Bumps [cloud.google.com/go/pubsub](https://github.com/googleapis/google-cloud-go) from 1.31.0 to 1.32.0. - [Release notes](https://github.com/googleapis/google-cloud-go/releases) - [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md) - [Commits](https://github.com/googleapis/google-cloud-go/compare/pubsub/v1.31.0...pubsub/v1.32.0) --- updated-dependencies: - dependency-name: cloud.google.com/go/pubsub dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Programmatically choose target Beam version for Automated Expansion Service tests (#27274) * Increase target Beam version for Automated Expansion Service testing * Automate selection of target JAR version Add explicit PortablePipelineOptions.enableHeapDumps to control heap dumping. (#26982) Finish Java Exception Sampling (#27257) * Finish Java Exception Sampling * wrong param name in comment * run tests * run tests * run tests [Python] Add note for using TF2 models with TensorFlow Hub (#27276) * add note for tf2 models * Update examples/notebooks/beam-ml/run_inference_with_tensorflow_hub.ipynb Co-authored-by: Rebecca Szper <98840847+rszper@users.noreply.github.com> * Update examples/notebooks/beam-ml/run_inference_with_tensorflow_hub.ipynb Co-authored-by: Rebecca Szper <98840847+rszper@users.noreply.github.com> --------- Co-authored-by: Rebecca Szper <98840847+rszper@users.noreply.github.com> Bump cloud.google.com/go/datastore from 1.11.0 to 1.12.0 in /sdks (#27281) Bumps [cloud.google.com/go/datastore](https://github.com/googleapis/google-cloud-go) from 1.11.0 to 1.12.0. - [Release notes](https://github.com/googleapis/google-cloud-go/releases) - [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/documentai/CHANGES.md) - [Commits](https://github.com/googleapis/google-cloud-go/compare/kms/v1.11.0...kms/v1.12.0) --- updated-dependencies: - dependency-name: cloud.google.com/go/datastore dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Add boilerplate Java21 security properties support to sdk containers (#26798) Fix cleanUpDockerJavaImages fail (#27287) Implement Exception Sampling in the Python SDK (#27280) * Python exception sampling implementation * add to cython def * add more cython defs * address comments * fix circular imports * linter * fix tests * remove print * Add traceback to exception * fix tests --------- Co-authored-by: Sam Rohde <srohde@google.com> Moving to 2.50.0-SNAPSHOT on master branch. Bump datastore-v1-proto-client dependency to match libraries-bom (#27285) Bump github.com/aws/aws-sdk-go-v2/feature/s3/manager in /sdks (#27295) Bumps [github.com/aws/aws-sdk-go-v2/feature/s3/manager](https://github.com/aws/aws-sdk-go-v2) from 1.11.70 to 1.11.71. - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases) - [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/CHANGELOG.md) - [Commits](https://github.com/aws/aws-sdk-go-v2/compare/feature/s3/manager/v1.11.70...feature/s3/manager/v1.11.71) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go-v2/feature/s3/manager dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Bump cloud.google.com/go/storage from 1.30.1 to 1.31.0 in /sdks (#27294) Bumps [cloud.google.com/go/storage](https://github.com/googleapis/google-cloud-go) from 1.30.1 to 1.31.0. - [Release notes](https://github.com/googleapis/google-cloud-go/releases) - [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md) - [Commits](https://github.com/googleapis/google-cloud-go/compare/pubsub/v1.30.1...pubsub/v1.31.0) --- updated-dependencies: - dependency-name: cloud.google.com/go/storage dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> bump govers to 1.20.5 (#27293) Simplify SchemaTransform API (#27202) * first batch: schematransform API, sql, debezium, fileIO, bigquery, bigtable, jdbc, singlestore * second batch: pubsub, pubsublite, spanner * third batch: kafka * fixes * fix missing override * bigtable write * spotless Update multi model notebook to remove workarounds (#27305) * Update multi model notebook to remove workarounds * Clean up + add conclusion Use requirements file as a constraint when building the expansion service container [#27207] Simplify Go quickstart. Add flag to do a docker system prune between docker push steps (#26967) * Add flag to do a docker system prune between docker push steps * Push to my docker image * order/prune python container pushes * Prune docker between pushes * Consolidate to loop * Temporarily point to my dockerhub to test/confirm * restore apache as dockerhub repo root * whitespace changed rerun logic to accomodate when job is not in check list (#27318) Populate getWorkStream latencies in dataflow streaming worker harness (#26085) Squash of two similar checks, not null check for "restriction.milisToRun" before unboxing the primitive (#27304) Co-authored-by: Anton_Shalkovich <anton_shalkovich@epam.com> Publish Docker artifacts from build_release_candidate.yml (#26950) * Update build_release_candidate.yml * rm maven settings * Install python * Run docker prune after each big push …

* Remove Beam Dependency Report code * Remove reference in README * Remove other references to the dependency job * More references

Remove Beam Dependency Report code

f8f1110

github-actions bot added the infra label May 31, 2023

jrmccluskey marked this pull request as ready for review June 1, 2023 17:09

damccorm reviewed Jun 1, 2023

View reviewed changes

Remove reference in README

40aa4b3

damccorm reviewed Jun 1, 2023

View reviewed changes

Remove other references to the dependency job

b74486f

github-actions bot added build python labels Jun 1, 2023

damccorm reviewed Jun 1, 2023

View reviewed changes

More references

1884c27

Abacn approved these changes Jun 6, 2023

View reviewed changes

jrmccluskey merged commit 5206952 into apache:master Jun 6, 2023

jrmccluskey deleted the noMoreEmail branch June 6, 2023 17:22

jrmccluskey mentioned this pull request Jun 6, 2023

Change queried target for Python Dependency Check #26897

Closed

3 tasks

cushon pushed a commit to cushon/beam that referenced this pull request May 24, 2024

Remove Beam Dependency Report code (apache#26954)

ace32d6

* Remove Beam Dependency Report code * Remove reference in README * Remove other references to the dependency job * More references

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Remove Beam Dependency Report code #26954

Remove Beam Dependency Report code #26954

jrmccluskey commented May 31, 2023

jrmccluskey commented Jun 1, 2023

github-actions bot commented Jun 1, 2023

damccorm left a comment

jrmccluskey commented Jun 1, 2023

damccorm Jun 1, 2023

damccorm Jun 1, 2023

jrmccluskey Jun 1, 2023

damccorm Jun 1, 2023

damccorm Jun 1, 2023

jrmccluskey Jun 1, 2023

damccorm Jun 1, 2023

jrmccluskey Jun 1, 2023

codecov bot commented Jun 1, 2023 •

edited

Loading

jrmccluskey commented Jun 6, 2023

jrmccluskey commented Jun 6, 2023

Abacn left a comment

Remove Beam Dependency Report code #26954

Remove Beam Dependency Report code #26954

Conversation

jrmccluskey commented May 31, 2023

GitHub Actions Tests Status (on master branch)

jrmccluskey commented Jun 1, 2023

github-actions bot commented Jun 1, 2023

damccorm left a comment

Choose a reason for hiding this comment

jrmccluskey commented Jun 1, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Jun 1, 2023 • edited Loading

Codecov Report

jrmccluskey commented Jun 6, 2023

jrmccluskey commented Jun 6, 2023

Abacn left a comment

Choose a reason for hiding this comment

codecov bot commented Jun 1, 2023 •

edited

Loading