[Failing Test]: beam/testing/passert Go Test flaky

[Abacn]

What happened?

This test has been flaky on Go Test GitHub Actions for a while. Example run: https://github.com/apache/beam/actions/runs/6426709641/job/17451278741

example error:

2023/10/06 02:25:55 ERROR process bundle failed for instruction inst004 using plan stage-007 : while executing Process for Plan[stage-007]:
1: DataSource[S[stage-007_source@localhost:42229], i0] Out:2 Coder:W;cwv_n9<bytes;c0>!GWC 
2: ParDo[passert.failIfBadEntries] Out:[] Sig: func([]uint8, func(*typex.T) bool, func(*typex.T) bool, func(*typex.T) bool) error, SideInputs: [SideInputAdapter[S[e8@localhost:42229], i1] - Coder W<string;c2>!GWC SideInputAdapter[S[e8@localhost:42229], i2] - Coder W<string;c2>!GWC SideInputAdapter[S[e8@localhost:42229], i3] - Coder W<string;c2>!GWC]
	caused by:
source failed
	caused by:
DoFn[UID:2, PID:passert.EqualsList/passert.failIfBadEntries, Name: github.com/apache/beam/sdks/v2/go/pkg/beam/testing/passert.failIfBadEntries] failed:
actual PCollection does not match expected values
=========
2 correct entries (present in both)
=========
1 unexpected entries (present in actual, missing in expected)
+++
example
=========
1 missing entries (missing in actual, present in expected)

Issue Failure

Failure: Test is flaky

Issue Priority

Priority: 1 (unhealthy code / failing or flaky postcommit so we cannot be sure the product is healthy)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[lostluck]

This is certainly a flake on Prism I wasn't aware of. Investigating.

[lostluck]

It's definitely a prism thing, and not this specific test. Recently there were other failures (which have been fixed) due to an uncompiled change failure change (unused import).

I see a few test timeouts.
I see a few "runner ran out of memory" (#28954 should help there) by properly caching go deps.
I see a couple where govulncheck is taking ages (this has now been removed, which will help).

One "send on a closed channel" which is definitely a race condition (and statistically likely to be the root cause of the timeouts).

https://github.com/apache/beam/actions/runs/6432180597/job/17466495105
https://github.com/apache/beam/actions/runs/6424305310/job/17444659174

[lostluck]

Scratch that. The caching and no more vuln check runs does seem to have speed things up from ~8-12 minutes for test runs to ~3. I'll take it.

[lostluck]

This seems to have been resolved between the various stability fixes to prism and the precommits.