Stop any ongoing cleanup on Job termination and on cleanup startup

[zimnx]

Description of your changes:

In case when Cleanup Job needs to be recreated by the Operator, the new
Job could complain about that it cannot execute cleanup operation, because there's
already one running.
Also when user would observe that maintenance operations have too big
impact on cluster performance he could decide to stop the cleanup.
Explicit stop is required as Scylla keeps cleaning even when API request
client disconnects.

In case when Cleanup Job was terminated ungracefully, a cleanup might
still be running in Scylla. As we can't keep track of it, a new one
needs to be started from the begining. To overcome an error returned by
Scylla API when cleanup is triggered and there's still one running, we
will stop all ongoing cleanups at the begining.

With this change all ongoing cleanup compactions are stopped when Cleanup Job terminates
due to received stop signal, and on each cleanup startup.

Which issue is resolved by this Pull Request:

Resolves #1343

[rzetelskik]

a few nits, other than that /lgtm

[tnozicka]

In case when Cleanup Job needs to be recreated by the Operator, the new Job could complain about that it cannot execute cleanup operation, because there's already one running.

What if the stop operation fails? (It's not retried, it may fail or the pod may terminate ungracefully.)
Is there an option to stop the old one when the new one is starting?

[zimnx]

/retest-required

[tnozicka]

/test images

[zimnx]

/retest-required

[tnozicka]

What if the stop operation fails? (It's not retried, it may fail or the pod may terminate ungracefully.)
Is there an option to stop the old one when the new one is starting?

@zimnx shouldn't the cancellation be retried with some backoff or better retried on next job start?

[zimnx]

shouldn't the cancellation be retried with some backoff

There's 1s backoff.

better retried on next job start?

Any argument behind why retrying on next job start would be better?

[tnozicka]

There's 1s backoff.

ok, I see there is now a backoff for 10s, thx

Any argument behind why retrying on next job start would be better?

not better, but in addition to cancellation on termination

• pods may terminate ungracefully, cleanup won't be stopped at all
• cleanup cancellation may fail and we can't afford to retry it forever because of terminationGracePeriod. Eventually, there is always a case where this could fail to cancel and something should likely handle it.

the new Job could complain about that it cannot execute cleanup operation, because there's already one running.

What exactly happens if the cancellation fails and a new job tries to start another cleanup while the old one is running? Is it just a warning, stuck until completion or the new one fails?

[zimnx]

What exactly happens if the cancellation fails and a new job tries to start another cleanup while the old one is running? Is it just a warning, stuck until completion or the new one fails?

ERROR 2023-08-21 09:26:15,917 [shard 0] api - Failed force_keyspace_cleanup of system_schema.table{name=columns, id=24101c25-a2ae-3af7-87c1-b40ee1aca33f}: std::runtime_error (cleanup request failed: there is an ongoing cleanup on system_schema.columns)

Which fails the request that triggers it. Job would be restarted - Job RestartPolicy is set to OnFailure - attempt to stop would be sent, if unsuccessful Job would spin until this "unstoppable" cleanup would finish, eventually triggering new one. In case when stop would be successful, new one should be triggered right after.

---

in addition to cancellation on termination

That would solve the ungraceful shutdown, I'll add it.

[tnozicka]

/approve
/lgtm

/hold
(for https://github.com/scylladb/scylla-operator/pull/1345/files#r1304437045)

[scylla-operator-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: tnozicka, zimnx

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [tnozicka,zimnx]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[tnozicka]

thanks for the updates - one pending question, feel free to /hold cancel if that's confirmed

[zimnx]

/hold cancel