[BEAM-14173] Fix Go Loadtests on Dataflow & partial fix for Flink

[lostluck]

Makes a minor breaking change to the SourceConfigs for the synthetic.Source testing io. The int fields are now int64 to become known types. In Go int is architecture dependent.

Flink & the Java runners don't appear to like having Schemas with "unknown logical types", which significantly reduces the value of logical types to "something known across all SDKs". Especially since in this instance when no runner side manipulation is required of them.

Dataflow loadtests were failing because the default container wasn't being found, but specifying the automatically built and pushed latest container like Python does resolves the issue.

PR also fixes an inconsistency in the jenkins README & display of one of the loadtest suites.

---

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

• Choose reviewer(s) and mention them in a comment (R: @username).
• Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
• Update CHANGES.md with noteworthy changes.
• If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

[lostluck]

Run Load Tests Go CoGBK Flink Batch

[lostluck]

Run Load Tests Go Combine Flink Batch

[lostluck]

Run Load Tests Go GBK Flink Batch

[lostluck]

Run Load Tests Go ParDo Flink Batch

[lostluck]

Run Load Tests Go SideInput Flink Batch

[codecov]

Codecov Report

Merging #17554 (a9e5b39) into master (017f846) will not change coverage.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##           master   #17554   +/-   ##
=======================================
  Coverage   73.88%   73.88%           
=======================================
  Files         691      691           
  Lines       91543    91543           
=======================================
  Hits        67641    67641           
  Misses      22669    22669           
  Partials     1233     1233           

Flag	Coverage Δ
go	50.14% <100.00%> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
sdks/go/pkg/beam/io/synthetic/source.go	64.58% <100.00%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 017f846...a9e5b39. Read the comment docs.

[lostluck]

Looks like the whole pipeline initialization set up has bit-rot while this has been broken. Likely easier to fix the Dataflow tests & their missing SDK image first.

[lostluck]

Run Load Tests Go SideInput Flink Batch

[lostluck]

Run Load Tests Go SideInput Dataflow Batch suite

[lostluck]

Run Load Tests Go SideInput Dataflow Batch

[lostluck]

Run Load Tests Go SideInput Dataflow Batch

[lostluck]

Run Seed Job

[lostluck]

Run Load Tests Go SideInput Dataflow Batch

[lostluck]

Run Load Tests Go SideInput Dataflow Batch

[lostluck]

Run Load Tests Go GBK Dataflow Batch

[lostluck]

Run Load Tests Go CoGBK Dataflow Batch

[lostluck]

Run Load Tests Go Combine Dataflow Batch

[lostluck]

Run Load Tests Go GBK Dataflow Batch

[lostluck]

Run Load Tests Go ParDo Dataflow Batch

[lostluck]

Run Seed Job Standalone

[lostluck]

Run Seed Job

[lostluck]

Run Load Tests Go ParDo Dataflow Batch

[lostluck]

Run Load Tests Go GBK Dataflow Batch

[lostluck]

Run Load Tests Go Combine Dataflow Batch

[lostluck]

Run Load Tests Go CoGBK Dataflow Batch

[lostluck]

Run Load Tests Go SideInput Dataflow Batch

[lostluck]

Run Load Tests Go SideInput Dataflow Batch

[lostluck]

Run Load Tests Go SideInput Dataflow Batch

[lostluck]

Run Load Tests Go GBK Dataflow Batch

[lostluck]

OK, leaving at demonstrating two of them. The runs can collide and the jobs are all passing, if not represented by the interrupted jenkins runs.

A Seed job would need to be run, before retyring to ensure the local changes are reflected by the runs.

[github-actions]

Assigning reviewers. If you would like to opt out of this review, comment assign to next reviewer:

R: @riteshghorse for label go.

Available commands:

• stop reviewer notifications - opt out of the automated review tooling
• remind me after tests pass - tag the comment author after tests pass
• waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

[lostluck]

Thanks!