[FLINK-5007] [checkpointing] Retain externalized checkpoint on suspension

[uce]

Handles graceful cluster shut down (non-HA) like cancellation and respects the configured clean up behaviour.

ExternalizedCheckpointCleanup.DELETE_ON_CANCELLATION => delete on suspension

ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION => retain on suspension

[uce]

@StephanEwen Do you have time to look at this? Currently, when externalized checkpoints are configured and the cluster shuts down via suspending all jobs, the externalized checkpoints are cleaned up. This PR proposes to handle suspension like a cancellation and respect the corresponding cleanup configuration, e.g. retain if RETAIN_ON_CANCELLATION and delete if DELETE_ON_CANCELLATION.

[StephanEwen]

A suspension is usually a master failure or loss of leadership. Suspending a job does not delete/remove any HA checkpoints.

The same should be the case for externalized checkpoints, in my opinion. Why should an externalized checkpoint be deleted in suspend, when a regular HA checkpoint is not?

[uce]

In HA mode, checkpoints are not deleted on suspension. This PR won't change that behaviour. It only affects non-HA behaviour.

Currently, the behaviour is to remove checkpoints on suspension, which is definitely a problem. But in non-HA mode suspension happens also for graceful shut down (for example when terminating a YARN session). Never deleting on suspend means that users who have DELETE_ON_CANCELLATION configured, will have externalized checkpoints lingering around when they shut down their non-HA cluster. That's why I thought it might be better to treat this the same as the retain/delete on cancellation configuration.

Does this make sense? For HA, these setting do not apply during suspension.

[StephanEwen]

Okay, understood, that makes sense.

+1 from my side