[BEAM-10955] Flink Java Runner test flake: Could not find Flink job #15127

benWize · 2021-07-05T15:44:45Z

A different approach from #15107 to fix this flaky test, FlinkSavepointTest in Java PreCommit

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Choose reviewer(s) and mention them in a comment (R: @username).
Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

`ValidatesRunner` compliance status (on master branch)

Lang	ULR	Samza	Twister2
Go	---	---	---
Java
Python	---	---	---
XLang		---	---

Examples testing status on various runners

Lang	ULR	Dataflow	Flink	Samza	Spark	Twister2
Go	---	---	---	---	---	---	---
Java	---		---	---	---	---	---
Python	---	---	---	---	---	---	---
XLang	---	---	---	---	---	---	---

Post-Commit SDK/Transform Integration Tests Status (on master branch)

Go	Java	Python

Pre-Commit Tests Status (on master branch)

---	Java	Python	Go	Website	Whitespace	Typescript
Non-portable
Portable	---			---	---	---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

…und flakiness

…testing

benWize · 2021-07-05T15:45:03Z

Run Java PreCommit

benWize · 2021-07-05T16:15:20Z

Run Java PreCommit

benWize · 2021-07-05T16:26:23Z

Run Java PreCommit

benWize · 2021-07-05T16:39:57Z

Run Java PreCommit

benWize · 2021-07-05T16:54:56Z

Run Java PreCommit

benWize · 2021-07-05T17:11:10Z

Run Java PreCommit

benWize · 2021-07-05T17:23:49Z

Run Java PreCommit

benWize · 2021-07-05T17:37:13Z

Run Java PreCommit

benWize · 2021-07-05T17:56:27Z

Consecutive PreCommit with FlinkSavePointTest passing

benWize · 2021-07-05T17:58:25Z

R: @ibzib
R: @tvalentyn
Would you help me to review this?

ibzib

Please fix checkstyle (:runners:flink:1.13:checkstyleTest)

ibzib · 2021-07-07T20:41:41Z

runners/flink/src/test/java/org/apache/beam/runners/flink/FlinkSavepointTest.java

@@ -159,6 +157,9 @@ private void runSavepointAndRestore(boolean isPortablePipeline) throws Exception
    // Initial parallelism
    options.setParallelism(2);
    options.setRunner(FlinkRunner.class);
+    // Enable checkpointing interval for streaming non portable pipeline to avoid


Why is this not a problem for portable pipelines?

I'm not sure about this, but when I set a checkpointing interval for a portable pipeline, it shows a timeout error like in https://ci-beam.apache.org/job/beam_PreCommit_Java_Phrase/3819/testReport/org.apache.beam.runners.flink/FlinkSavepointTest/testSavepointRestorePortable_2/.
The reason behind this fix is to enable restart after some job failure.
When this test fails, continuously shows the error: "Recovery is suppressed by NoRestartBackoffTimeStrategy" like in https://scans.gradle.com/s/n2coqujl4jc7i/tests/:runners:flink:1.13:test/org.apache.beam.runners.flink.FlinkSavepointTest/testSavepointRestoreLegacy?top-execution=1.

I'm not sure about this, but when I set a checkpointing interval for a portable pipeline, it shows a timeout error like in https://ci-beam.apache.org/job/beam_PreCommit_Java_Phrase/3819/testReport/org.apache.beam.runners.flink/FlinkSavepointTest/testSavepointRestorePortable/.

I'm not sure which error you are talking about? If the test passed, it's likely it's benign.

The reason behind this fix is to enable restart after some job failure.
When this test fails, continuously shows the error: "Recovery is suppressed by NoRestartBackoffTimeStrategy" like in https://scans.gradle.com/s/n2coqujl4jc7i/tests/:runners:flink:1.13:test/org.apache.beam.runners.flink.FlinkSavepointTest/testSavepointRestoreLegacy?top-execution=1.

Thanks for getting the build scan. It looks like something is going wrong while taking the savepoint. It looks like it could be a real bug, so let's wait to merge this until we are sure that's not the case.

Caused by: java.lang.Exception: Could not materialize checkpoint 2 for operator VerificationStage/ParMultiDo(Anonymous) (1/2)#0. | | at org.apache.flink.streaming.runtime.tasks.AsyncCheckpointRunnable.handleExecutionException(AsyncCheckpointRunnable.java:257) | | ... 4 more | | Caused by: java.lang.IllegalArgumentException | | at org.apache.flink.util.Preconditions.checkArgument(Preconditions.java:122) | | at org.apache.flink.runtime.checkpoint.CheckpointMetrics.<init>(CheckpointMetrics.java:74) | | at org.apache.flink.runtime.checkpoint.CheckpointMetricsBuilder.build(CheckpointMetricsBuilder.java:135) | | at org.apache.flink.streaming.runtime.tasks.AsyncCheckpointRunnable.reportCompletedSnapshotStates(AsyncCheckpointRunnable.java:206) | | at org.apache.flink.streaming.runtime.tasks.AsyncCheckpointRunnable.run(AsyncCheckpointRunnable.java:158) | | ... 3 more

It looks like the failed precondition is checking alignmentDurationNanos. I'm not sure however what the unacceptable value is, nor where it is coming from. https://github.com/apache/flink/blob/3909c9f0a11e8b38b264db9e7716fb41e75cc524/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointMetrics.java#L74

benWize · 2021-07-07T23:13:07Z

Please fix checkstyle (:runners:flink:1.13:checkstyleTest)

Fixed

benWize · 2021-07-07T23:15:54Z

runners/flink/src/test/java/org/apache/beam/runners/flink/FlinkSavepointTest.java

@@ -194,7 +196,7 @@ private void runSavepointAndRestore(boolean isPortablePipeline) throws Exception
  private JobID executeLegacy(Pipeline pipeline) throws Exception {
    JobGraph jobGraph = getJobGraph(pipeline);
    flinkCluster.submitJob(jobGraph).get();
-    return jobGraph.getJobID();
+    return waitForJobToBeReady();


I added this line here, maybe if this PR is accepted and merged we can close #15119

aaltay · 2021-07-16T03:37:23Z

What is the next step on this PR?

ibzib · 2021-07-20T17:56:48Z

What is the next step on this PR?

I'm pretty sure the true cause of the flake is a Flink issue (FLINK-23201), so it is fine to ignore the test until we upgrade to the Flink patch releases containing the fix.

benWize · 2021-07-20T18:15:44Z

What is the next step on this PR?

I'm pretty sure the true cause of the flake is a Flink issue (FLINK-23201), so it is fine to ignore the test until we upgrade to the Flink patch releases containing the fix.

Is it ok to close this PR then?

ibzib · 2021-07-20T18:26:57Z

What is the next step on this PR?

I'm pretty sure the true cause of the flake is a Flink issue (FLINK-23201), so it is fine to ignore the test until we upgrade to the Flink patch releases containing the fix.

Is it ok to close this PR then?

Yes.

benWize added 5 commits June 30, 2021 10:44

[BEAM-10955] Testing could not find Flink job

d7a416d

[BEAM-10955] Add waitJobToBeReady call, test fix for Flink job not fo…

69874ee

…und flakiness

[BEAM-10955] Remove unused import

d04d6c4

[BEAM-10955] Discard changes in website folder, remove loop used for …

1872ae3

…testing

[BEAM-10955] Enable checkpointing interval

e60e2e1

benWize added 2 commits July 5, 2021 11:09

[BEAM-10955] Add checkpointing interval only for non portable pipeline

f875363

[BEAM-10955] Spotless apply

5482a08

benWize marked this pull request as ready for review July 5, 2021 17:57

benWize changed the title ~~[WIP] [BEAM-10955] Flink Java Runner test flake: Could not find Flink job~~ [BEAM-10955] Flink Java Runner test flake: Could not find Flink job Jul 5, 2021

ibzib reviewed Jul 7, 2021

View reviewed changes

[BEAM-10955] Fix style

4321eb7

benWize commented Jul 7, 2021

View reviewed changes

benWize closed this Jul 20, 2021

benWize deleted the BEAM-10955-checkpointing-fix branch August 5, 2021 22:03

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BEAM-10955] Flink Java Runner test flake: Could not find Flink job #15127

[BEAM-10955] Flink Java Runner test flake: Could not find Flink job #15127

benWize commented Jul 5, 2021

benWize commented Jul 5, 2021

benWize commented Jul 5, 2021

benWize commented Jul 5, 2021

benWize commented Jul 5, 2021

benWize commented Jul 5, 2021

benWize commented Jul 5, 2021

benWize commented Jul 5, 2021

benWize commented Jul 5, 2021

benWize commented Jul 5, 2021

benWize commented Jul 5, 2021

ibzib left a comment

ibzib Jul 7, 2021

benWize Jul 7, 2021 •

edited

Loading

ibzib Jul 8, 2021

benWize commented Jul 7, 2021

benWize Jul 7, 2021

aaltay commented Jul 16, 2021

ibzib commented Jul 20, 2021

benWize commented Jul 20, 2021 •

edited

Loading

ibzib commented Jul 20, 2021

[BEAM-10955] Flink Java Runner test flake: Could not find Flink job #15127

[BEAM-10955] Flink Java Runner test flake: Could not find Flink job #15127

Conversation

benWize commented Jul 5, 2021

ValidatesRunner compliance status (on master branch)

Examples testing status on various runners

Post-Commit SDK/Transform Integration Tests Status (on master branch)

Pre-Commit Tests Status (on master branch)

GitHub Actions Tests Status (on master branch)

benWize commented Jul 5, 2021

benWize commented Jul 5, 2021

benWize commented Jul 5, 2021

benWize commented Jul 5, 2021

benWize commented Jul 5, 2021

benWize commented Jul 5, 2021

benWize commented Jul 5, 2021

benWize commented Jul 5, 2021

benWize commented Jul 5, 2021

benWize commented Jul 5, 2021

ibzib left a comment

Choose a reason for hiding this comment

ibzib Jul 7, 2021

Choose a reason for hiding this comment

benWize Jul 7, 2021 • edited Loading

Choose a reason for hiding this comment

ibzib Jul 8, 2021

Choose a reason for hiding this comment

benWize commented Jul 7, 2021

benWize Jul 7, 2021

Choose a reason for hiding this comment

aaltay commented Jul 16, 2021

ibzib commented Jul 20, 2021

benWize commented Jul 20, 2021 • edited Loading

ibzib commented Jul 20, 2021

`ValidatesRunner` compliance status (on master branch)

benWize Jul 7, 2021 •

edited

Loading

benWize commented Jul 20, 2021 •

edited

Loading