Add backups with Kubernetes CronJobs

[metalmatze]

On one of my projects I've created a Kubernetes CronJob to automatically backup databases to S3-compatible object storage (with minio).
Going forward this should become a part of kube-cockroachdb.

We need to parameterize a few things, like schedule and secrets, to make it really work for most people.

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: cockroachdb-backup
  namespace: app
spec:
  schedule: "0 */12 * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: cockroachdb
            image: cockroachdb/cockroach:v20.1.0
            command:
            - /bin/bash
            - -ecx
            - exec /cockroach/cockroach dump --url $DSN app > /backups/app.sql
            env:
              - name: DSN
                valueFrom:
                  secretKeyRef:
                    key: DSN
                    name: app
            volumeMounts:
              - name: backups
                mountPath: /backups
          - name: minio
            image: minio/mc:RELEASE.2020-04-04T05-28-55Z
            command:
            - /bin/sh
            - -c
            - sleep 10 && mc cp /backups/app.sql app-$(date +%Y-%m-%d-%H-%M-%S).sql
            env:
              - name: MC_HOST_digitalocean
                value: 'https://XXX@ams3.digitaloceanspaces.com'
            volumeMounts:
              - name: backups
                mountPath: /backups
          volumes:
            - name: backups
              emptyDir: {}
          restartPolicy: OnFailure

Please add a reaction to indicate you want this feature and leave a comment for other requirements around this feature.

[brancz]

Cockroachdb 20.2 was released, which ships their previously only in the enterprise version backup and restore functionality that is directly built into it. It probably makes sense to use that well-hardened feature rather than something somewhat home-brewed.

[brancz]

https://www.cockroachlabs.com/docs/v20.2/take-full-and-incremental-backups#perform-core-backup-and-restore

[metalmatze]

I finally looked at these docs.
It seems that it still needs to be triggered from the outside (which is fine). Therefore the CronJob is still necessary, only the minio sidecar can disappear and it can be handled by CockroachDB itself.

[brancz]

Awesome!

[brancz]

Actually I read https://www.cockroachlabs.com/docs/v20.2/take-full-and-incremental-backups#automated-full-backups differently

[metalmatze]

Ah, seems like I missed that detail. Honestly, I'm not sure I want to have that part of the CRD in the end. 🤔 In the end, the question is how much overlap there should be between scheduling and scaling and actual database operation.

[brancz]

Yeah I agree, especially with this feature built into CDB like this I don't think it makes sense for this to be part of the CRD. I don't see what kube-cockroachdb would even do here.