Give possibility for restoring DC using mapping sourceDC -> destinationDC

[karol-kokoszka]

#3871

Right now, there is no option in the Scylla Manager restore task to restore just a single data center (DC) from the backup location. This could lead to problematic situations, particularly when:

Encryption at Rest (EaR) is enabled,
Two DCs use different encryption keys,
Encryption keys are stored in different cloud regions, and
There is only one backup location available.
To address this, we would need to make the encryption keys multi-regional to facilitate the restoration process in such scenarios.

The location flag may not be very intuitive, as the [dc] part defines the destination DC, not the source DC data. We need to discuss during the manager planning to determine if a new flag specifying the source is necessary.
If we can restore just a single DC, then we can restore DC by DC, avoiding the need to create multi-regional keys.

(cc: @tzach)

[tzach]

To address this, we would need to make the encryption keys multi-regional to facilitate the restoration process in such scenarios.

Agree, but how this is a Scylla Manager issue to fix?

[karol-kokoszka]

We could potentially address the problem by allowing to restore just a single DC from the location bucket.
It's something what we doesn't support at the moment (possibly by a mistake).

[rayakurl]

@tzach - we need a resolution. for now almost all sct tests are failing since they are multi DC. We will add a couple of pipelines for a single DC + encryption but are are disabling the multi DC jobs as they constantly failing. @mikliapko as discussed, please create a task for the new pipelines. and disable the multi DC ones for now. Thanks

[karol-kokoszka]

grooming notes

The initial idea is to add new flag to the restore CLI, so that it's possible to define the origin DC from the backup location.
Then, data from this DC is going to be restored to specified destination.

@mikliapko SCT will have to be updated to test the scenario with restoring single DC.

[Michal-Leszczynski]

The initial idea is to add new flag to the restore CLI, so that it's possible to define the origin DC from the backup location.
Then, data from this DC is going to be restored to specified destination.

After giving it some more thought, I wouldn't recommend adding it in such way.
The need for this feature raised from #3871, where it could be used to restore DC by DC.
This is problematic, as restore task does not only download and load&stream the data, but it also:

• disables and enables tombstone_gc
• drops and creates views
• runs repair

So running many restore tasks, one by one, DC by DC, would result in lots of redundant work.
Also, I could theoretically (not sure about that) lead to data resurrection, as tombstone_gc would be enabled in between DC restorations.
Not to mention, that it would be user responsibility to remember about all DCs from the backup to be restored.

A better idea could be to extend restore with a flag like --dc-mapping (string -> list of strings).
This would allow user to specify which dc from the backup should be restored by which dcs in restored cluster.
It has a few benefits;

• makes it possible to solve Manager restore error (kms_error AccessDeniedException) for multiDC cluster with EAR enabled #3871
• allows for tuning up batching process by reducing node - backup location latencies in multi-location setting
• it could be used during restore schema in order to modify CREATE KEYSPACE statements containing no longer existing dcs (Make it possible to restore schema into a different DC setting #4049)

[mikliapko]

A better idea could be to extend restore with a flag like --dc-mapping (string -> list of strings).

@Michal-Leszczynski
When it is ready, could you please then provide an example of input for this flag.
I will switch some of our SCT tests back to run on multiDC cluster.