[FLINK-31803] Harden UpdateJobResourceRequirementsRecoveryITCase.

[dmvk]

This fixes a race condition where HA data might have been accidentally cleaned up due to job transition to the terminal state.

https://issues.apache.org/jira/browse/FLINK-31803

[flinkbot]

CI report:

• f54c5e3 Azure: SUCCESS

Bot commands

The @flinkbot bot supports the following commands:

• @flinkbot run azure re-run the last Azure build

[zentol]

Which terminal state did the job enter, and why?

Couldn't the test still fail if the update call is made while a restarted job is still initializing?

[zentol]

I don't understand this change.

[dmvk]

This is meant in combination with:

        assumeThat(ClusterOptions.isAdaptiveSchedulerEnabled(configuration)).isTrue();

which ensures this is already set

[dmvk]

The main idea is to skip this test when smoke testing the default scheduler, because it can run for > 10s

[zentol]

I'd just consider this a test for the adaptive scheduler, and we don't categorically skip those in PRs.

If test times are a concern, well then those should be addressed anyway.

[dmvk]

I'd just consider this a test for the adaptive scheduler, and we don't categorically skip those in PRs.

Fixed.

If test times are a concern, well then those should be addressed anyway.

The proper fix would be to have a real HA setup for testing that doesn't require running Zookeeper, which is out of scope for now 😢

[dmvk]

Which terminal state did the job enter, and why?

FAILED because during TM disconnect, the task failed, and the NoRestartStrategy tore the job down.

Couldn't the test still fail if the update call is made while a restarted job is still initializing?

I think this is prevented by calling waitUntilJobInitializationFinished before the update, or am I missing something?

[dmvk]

Couldn't the test still fail if the update call is made while a restarted job is still initializing?

We don't make any more updates after spinning up the 2nd cluster. The job restart this PR refers to is caused by race conditions during closeAsyncWithoutCleaningHighAvailabilityData which happens after the update.

[zentol]

FAILED because during TM disconnect, the task failed, and the NoRestartStrategy tore the job down.

How does the restart strategy fix that? If the TM disconnects (which I assume happens due to the shutdown), surely the job wont reach a running state again.

Ah

[dmvk]

@zentol all comments should be addressed, PTAL