Schedule and replicate bootstrap daily backup #3557

alexandre-allard · 2021-10-06T07:36:23Z

Component: salt, backup

Context:
Actually we do backups only when performing an action on the cluster (bootstrap, upgrade, downgrade and restore) and we keep this backup locally.
It means that if we do not do any actions for a long time, the latest backup will be very old if we need it at some point.
Also, since the backup is not copied to other nodes, if we really lose the bootstrap node 💥 we will be unable to restore a new one.

Summary:
Schedule daily backups on the bootstrap node and copy this archive on the master nodes.

Acceptance criteria:

NicolasT · 2021-10-06T08:09:07Z

Instead of making a backup in one place, then replicating the backup(s) to all master nodes, would it make sense instead to create a (local) backup on every master node? What are these nodes missing in order to generate such backup? Basically: which data is currently not present on non-bootstrap master nodes for them to be able to become bootstrap? I guess mainly CA things?

Could we instead work on that, which would also further lead towards 'Bootstrap HA'?

TeddyAndrieux · 2021-10-06T09:08:01Z

Instead of making a backup in one place, then replicating the backup(s) to all master nodes, would it make sense instead to create a (local) backup on every master node? What are these nodes missing in order to generate such backup? Basically: which data is currently not present on non-bootstrap master nodes for them to be able to become bootstrap? I guess mainly CA things?

Could we instead work on that, which would also further lead towards 'Bootstrap HA'?

We also miss the configuration files (/etc/metalk8s/{bootstrap,solutions}.yaml).
I agree it could be a better approach to start thinking about a way to have everything available on "some nodes" but it's not as simple as for CA, that shouldn't change really often.

🤔
But maybe the way to go should be a Job (or a script) running on the master nodes that generate the backup, that today retrieve some needed information from the bootstrap node (so it could be, today, a Job that needs the bootstrap node to be up even if it generates the backup locally on every master node)

alexandre-allard · 2021-10-06T09:08:12Z

@NicolasT Yes, it is mainly the certificates and metalk8s/solutions configuration files.
We wanted to work on that, this is supposed to be a first iteration to have this feature quickly and gives us time to think on how to achieve that, with a proper design.

TeddyAndrieux

Few comments
I think you should add a changelog entry and btw I'm not sure it should be in 2.10

TeddyAndrieux · 2021-10-08T13:33:56Z

pillar/metalk8s/roles/ca.sls

@@ -23,6 +23,10 @@ mine_functions:
    - mine_function: hashutil.base64_encodefile
    - /etc/metalk8s/pki/nginx-ingress/ca.crt

+  backup_ca_b64:


It's a bit weird to me to have a "backup" CA but ... I have no better idea in mind, maybe just rename everywhere so that we know that it's "backup_server" CA

Btw wondering if we really need a CA for this backup server ? shouldn't we use just a self-signed server certificate ? 🤔

Maybe it would be sufficient, I don't know.

salt/_modules/metalk8s.py

salt/metalk8s/backup/certs/ca.sls

scripts/backup.sh.in

alexandre-allard · 2021-10-11T09:08:41Z

/reset

alexandre-allard · 2021-10-12T07:40:03Z

/approve

scripts/upgrade.sh.in

salt/metalk8s/backup/deployed/secret-credentials.sls

This HTTP server is used to serve the backups generated on the bootstrap node.

TeddyAndrieux

Let's go for it

bert-e · 2021-10-14T14:07:11Z

Build failed

The build for commit did not succeed in branch w/2.11/improvement/schedule-bootstrap-backup.

The following options are set: approve

bert-e · 2021-10-14T14:11:27Z

Build failed

The build for commit did not succeed in branch improvement/schedule-bootstrap-backup.

The following options are set: approve

bert-e · 2021-10-14T22:32:26Z

In the queue

The changeset has received all authorizations and has been added to the
relevant queue(s). The queue(s) will be merged in the target development
branch(es) as soon as builds have passed.

The changeset will be merged in:

✔️ development/2.10
✔️ development/2.11

The following branches will NOT be impacted:

development/2.0
development/2.1
development/2.2
development/2.3
development/2.4
development/2.5
development/2.6
development/2.7
development/2.8
development/2.9

There is no action required on your side. You will be notified here once
the changeset has been merged. In the unlikely event that the changeset
fails permanently on the queue, a member of the admin team will
contact you to help resolve the matter.

IMPORTANT

Please do not attempt to modify this pull request.

Any commit you add on the source branch will trigger a new cycle after the
current queue is merged.
Any commit you add on one of the integration branches will be lost.

If you need this pull request to be removed from the queue, please contact a
member of the admin team now.

The following options are set: approve

bert-e · 2021-10-14T23:53:20Z

I have successfully merged the changeset of this pull request
into targetted development branches:

✔️ development/2.10
✔️ development/2.11

The following branches have NOT changed:

development/2.0
development/2.1
development/2.2
development/2.3
development/2.4
development/2.5
development/2.6
development/2.7
development/2.8
development/2.9

Please check the status of the associated issue None.

Goodbye alexandre-allard-scality.