[Failing Test]: PubsubReadIT timeout pollForResultForDuration

[Abacn]

What happened?

happens here: https://github.com/apache/beam/blob/7890bdcee843851ae32354df051f8b8abec12bc6/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubReadIT.java#L85C5-L85C5

Error Message
java.lang.AssertionError: Did not receive signal on projects/apache-beam-testing/subscriptions/start-subscription--2539694162040025434 in 300s
Stacktrace
java.lang.AssertionError: Did not receive signal on projects/apache-beam-testing/subscriptions/start-subscription--2539694162040025434 in 300s
	at org.apache.beam.sdk.io.gcp.pubsub.TestPubsubSignal.pollForResultForDuration(TestPubsubSignal.java:334)
	at org.apache.beam.sdk.io.gcp.pubsub.TestPubsubSignal.lambda$waitForStart$4(TestPubsubSignal.java:276)
	at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Suppliers$NonSerializableMemoizingSupplier.get(Suppliers.java:167)
	at org.apache.beam.sdk.io.gcp.pubsub.PubsubReadIT.testReadPubsubMessageId(PubsubReadIT.java:85)
...
at org.apache.beam.sdk.testing.TestPipeline$1.evaluate(TestPipeline.java:323)
	at org.apache.beam.sdk.io.gcp.pubsub.TestPubsubSignal$1.evaluate(TestPubsubSignal.java:131)

Pipeline startup usually takes 3.5 min (and could be longer). The timeout is 5 min. This pipeline reads from public data (projects/pubsub-public-data/topics/taxirides-realtime) Could it be that no message between pipeline startup and timeout and causing flaky test?

Issue Failure

Failure: Test is flaky

Issue Priority

Priority: 1 (unhealthy code / failing or flaky postcommit so we cannot be sure the product is healthy)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[kennknowles]

Is it flaky and not perma-red? If you bump the timeout does it deflake it?

[Abacn]

It's flaky. Bump the timeout sure will decrease the flakiness, but only adds some probability. To completely deflaky it we should probably change the topic to one that reliably publishes message with certain frequency.

[Abacn]

also happens for PubsubLite (org.apache.beam.sdk.io.gcp.pubsublite.ReadWriteIT.testReadWrite - test same ting): https://ci-beam.apache.org/view/PostCommit/job/beam_PostCommit_Java_DataflowV2/3275/testReport/junit/org.apache.beam.sdk.io.gcp.pubsublite/ReadWriteIT/testReadWrite/