Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
136 changes: 79 additions & 57 deletions docs/backups-scheduled.md
Original file line number Diff line number Diff line change
@@ -1,38 +1,34 @@
# Making scheduled backups


Backups schedule is defined in the `backup` section of the Custom
Resource and can be configured via the [deploy/cr.yaml :octicons-link-external-16:](https://github.com/percona/percona-server-mysql-operator/blob/v{{release}}/deploy/cr.yaml)
file.
Scheduled backups run automatically at times you specify. You configure them in the `backup` section of your Custom Resource using the [deploy/cr.yaml :octicons-link-external-16:](https://github.com/percona/percona-server-mysql-operator/blob/v{{release}}/deploy/cr.yaml) file.

1. The `backup.storages` subsection should contain at least one [configured storage](backups-storage.md).
Before setting up scheduled backups, make sure you have at least one [configured storage](backups-storage.md) in the `backup.storages` subsection.

2. The `backup.schedule` subsection allows to actually schedule backups:
## Configure a backup schedule

* set the `backup.schedule.name` key to some arbitrary backup name (this name
will be needed later to [restore the backup](backups-restore.md)).
To schedule backups, specify the following configuration in your Custom Resource:

* specify the `backup.schedule.schedule` option with the desired backup
schedule in [crontab format :octicons-link-external-16:](https://en.wikipedia.org/wiki/Cron).
1. Enable backup by setting `backup.enabled` to `true`.
2. Add entries to the `backup.schedule` subsection in your Custom Resource. Each schedule entry requires the following:

* set the `backup.schedule.storageName` key to the name of your [already configured storage](backups-storage.md).
* `name` - a unique name for this backup schedule
* `schedule` - the backup schedule in [crontab format :octicons-link-external-16:](https://en.wikipedia.org/wiki/Cron). For example, `"0 0 * * 6"` runs every Saturday at midnight.
* `storageName` - the name of your [configured storage](backups-storage.md) where backups will be stored
* `keep` (optional) - the number of backups to keep in storage. Older backups are automatically deleted when this limit is reached.

* you can optionally set the `backup.schedule.keep` key to the number of
backups which should be kept in the storage.

Here is an example of the `deploy/cr.yaml` with a scheduled Saturday night
backup kept on the Amazon S3 storage:
Here's an example configuration that creates a backup every Saturday night at midnight and keeps the last 3 backups:

```yaml
...
backup:
enabled: true
storages:
s3-us-west:
type: s3
s3:
bucket: S3-BACKUP-BUCKET-NAME-HERE
region: us-west-2
credentialsSecret: my-cluster-name-backup-s3
credentialsSecret: ps-cluster1-s3-credentials
schedule:
- name: "sat-night-backup"
schedule: "0 0 * * 6"
Expand All @@ -43,56 +39,82 @@ backup:

## Managing multiple backup schedules in the same storage

You can define multiple backup schedules to meet different recovery and compliance needs. For example, create daily backups for quick recovery from recent changes and monthly backups for long-term retention or audit purposes.

When using the same storage location for multiple schedules, be aware of these important considerations:
You can define multiple backup schedules to meet different recovery and compliance needs. For example, you might want daily backups for quick recovery from recent changes and monthly backups for long-term retention or audit purposes.

1. **Retention policy conflicts.** The Operator only applies retention policies to the first schedule in your configuration. For example, if you set daily backups to keep 5 copies and monthly backups to keep 3 copies, the Operator will only keep 5 total backups in storage, not 8 as you might expect. However, all backup objects will still appear in `kubectl get ps-backup` output.
When you use the same storage location for multiple schedules, be aware of these important limitations:

2. **Concurrent backup conflicts.** When multiple schedules run simultaneously and write to the same storage path, backups can overwrite each other, resulting in incomplete or corrupted data.
1. **Retention policy conflicts**: The Operator only applies retention policies to the first schedule in your configuration. For example, if you set daily backups to keep 5 copies and monthly backups to keep 3 copies, the Operator will only keep 5 total backups in storage, not 8 as you might expect. However, all backup objects will still appear in `kubectl get ps-backup` output.

To avoid these issues and ensure each schedule maintains its own retention policy, configure separate storage locations for each schedule.
2. **Concurrent backup conflicts**: When multiple schedules run at the same time and write to the same storage path, backups can overwrite each other, resulting in incomplete or corrupted data.

The configuration steps are the following:
To avoid these issues and ensure each schedule maintains its own retention policy, configure separate storage locations for each schedule. Here's how:

1. Create separate storage configurations in your Custom Resource with different names
2. Specify unique buckets or prefixes for each storage configuration
3. Assign different storage names to specific schedules

Here is the example configuration:

```yaml
storages:
minio1:
type: s3
s3:
credentialsSecret: minio-secret
bucket: my-bucket
prefix: minio1
endpointUrl: "http://minio.minio.svc.cluster.local:9000/"
verifyTLS: false
minio2:
type: s3
s3:
credentialsSecret: minio-secret
bucket: my-bucket
prefix: minio2
endpointUrl: "http://minio.minio.svc.cluster.local:9000/"
verifyTLS: false

....

backup:
schedule:
- name: "daily-backup"
schedule: "0 2 * * *"
keep: 2
storageName: minio1
- name: "monthly-backup"
schedule: "0 2 1 * *"
keep: 5
storageName: minio2
Here's an example configuration that uses separate storage locations for daily and monthly backups:

```yaml
storages:
minio1:
type: s3
s3:
credentialsSecret: minio-secret
bucket: my-bucket
prefix: minio1
endpointUrl: "http://minio.minio.svc.cluster.local:9000/"
verifyTLS: false
minio2:
type: s3
s3:
credentialsSecret: minio-secret
bucket: my-bucket
prefix: minio2
endpointUrl: "http://minio.minio.svc.cluster.local:9000/"
verifyTLS: false

....

backup:
schedule:
- name: "daily-backup"
schedule: "0 2 * * *"
keep: 2
storageName: minio1
- name: "monthly-backup"
schedule: "0 2 1 * *"
keep: 5
storageName: minio2
```

In this example, daily backups are stored in `minio1` and monthly backups are stored in `minio2`, each with their own retention policy.

## Monitoring scheduled backups

When a scheduled backup runs, the Operator creates a Kubernetes Job to execute the backup. You can view these jobs to monitor backup execution:

```bash
kubectl get jobs -n <namespace>
```

??? example "Example output"

```{.text .no-copy}
NAME COMPLETIONS DURATION AGE
xb-cron-ps-cluster2-gcp-20251107152047-9mgo5-gcp 1/1 9s 21s
```

To find all backups created by a specific schedule, use the `percona.com/backup-ancestor` label. The label format is `percona.com/backup-ancestor: <hash>-<schedule-name>`, where `<hash>` is a unique identifier and `<schedule-name>` is the name you specified in your schedule configuration.

For example, to find all backups created by the `sat-night-backup` schedule:

```bash
kubectl get ps-backup -l percona.com/backup-ancestor -n <namespace>
```

Or to filter for a specific schedule:

```bash
kubectl get ps-backup -l percona.com/backup-ancestor=0ed1d-sat-night-backup -n <namespace>
```