[FLINK-35801] Fix SnapshotFileMergingCompatibilityITCase, wait for file deletion before quit

[Zakelly]

What is the purpose of the change

This PR fixes the unstable test SnapshotFileMergingCompatibilityITCase. There is a problem with the test code itself. The test verifies the cp files are cleaned as expected, which depends on an async cleaner (the job io thread). While the mini-cluster quit, the async threads may not have finished their jobs before termination.

It is better to try multiple rounds AZP to verify this fix.

Brief change log

Only SnapshotFileMergingCompatibilityITCase. Make it wait for file deletion before cluster termination.

Verifying this change

Does this pull request potentially affect one of the following parts:

• Dependencies (does it add or upgrade a dependency): no
• The public API, i.e., is any changed class annotated with @Public(Evolving): no
• The serializers: no
• The runtime per-record code paths (performance sensitive): no
• Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
• The S3 file system connector: no

Documentation

• Does this pull request introduce a new feature? no
• If yes, how is the feature documented? not applicable

[flinkbot]

CI report:

• 03f2c07 Azure: SUCCESS

Bot commands

The @flinkbot bot supports the following commands:

• @flinkbot run azure re-run the last Azure build

[fredia]

@Zakelly Thanks for the quick fix, overall LGTM.

[fredia]

If the timeout is reached and the files are not deleted, will this check failed?
If restoring from file-merging disabled to file-merging disabled, is it okay to skip this check?

[Zakelly]

If the timeout is reached and the files are not deleted, will this check failed?

Yes.

If restoring from file-merging disabled to file-merging disabled, is it okay to skip this check?

In this case, we are not testing the file-merging, but we could check the file existence by the way.

[Zakelly]

I don't think the CI failure is related with file-merging. It happens when file-merging is disabled. I'm investigating the potential file leak.

Update: Well there is an issue with the test code.

[Zakelly]

I have run the test with the last commit for 200 times. No failure.

[fredia]

Thanks for investigating and updating, let's wait CI green.