[FLINK-31133][tests] Fix timeouts in PartiallyFinishedSourcesITCase

[rkhachatryan]

If a checkpoint fails for some reason, PartiallyFinishedSourcesITCase might hang (and indirectly affect other tests).

There are two issues in case of a checkpoint failure:

1. FAIL command might be dispatched to the source task that's already finished execution
2. waiting for the failover times out, but it then waits indefinitely to obtain job status result

Mostly happens in 1.15, because in later versions have increased timeouts/attempts to upload state.

This PR fixes the above root causes of test hanging.

[flinkbot]

CI report:

• a3c4e0c Azure: SUCCESS

Bot commands

The @flinkbot bot supports the following commands:

• @flinkbot run azure re-run the last Azure build

[rkhachatryan]

@gaoyunhaii would you be able to take a look at this PR?

[dmvk]

Thanks for the patch Roman; this reads very sane! (though I'm not an expert in this part of Flink).

Apart from inline questions:

1. The first commit should be rephrased

Prevent timeouts when waiting for a status of a …

->

Prevent timeouts when waiting for a result of a …

--

2. For the second commit

Otherwise, any command with SINGLE_SUBTASK scope might be dispatched to a finished source.
This will result in a timeout while waiting for this command to be executed.

Can you please point me to where precisely the timeout is supposed to happen? As far as I can tell, on the source side, we'd only add it to the queue (there will most likely be something on the executor side that I'm missing).

[rkhachatryan]

Thanks for the review @dmvk!

I've updated both commit messages and marked scheduledCommands as volatile.

For the second commit

Otherwise, any command with SINGLE_SUBTASK scope might be dispatched to a finished source.
This will result in a timeout while waiting for this command to be executed.

Can you please point me to where precisely the timeout is supposed to happen? As far as I can tell, on the source side, > we'd only add it to the queue (there will most likely be something on the executor side that I'm missing).

Sure, the timeout happens in TestJobExecutor.waitForFailover.

[dmvk]

LGTM, good job 👍 Can you, please squash the hotfixes into the original two commits before merging?

[rkhachatryan]

Thanks for the review @dmvk!
I've squashed the commits, will merge the PR once the build succeeds.