Add storage bucket backup

[maveonair]

This pull request add backup support for storage buckets from #241.

Description

Export

CLI

$ incus storage bucket export
Description:
  Export storage buckets as tarball.

Usage:
  incus storage bucket export [<remote>:]<pool> <bucket> [<path>] [flags]

Examples:
  incus storage bucket default b1
      Download a backup tarball of the b1 storage bucket.

Flags:
      --compression   Define a compression algorithm: for backup or none
      --target        Cluster member name

Global Flags:
      --debug          Show all debug messages
      --force-local    Force using the local unix socket
  -h, --help           Print help
      --project        Override the source project
  -q, --quiet          Don't show progress information
      --sub-commands   Use with help or --help to view sub-commands
  -v, --verbose        Show all information messages
      --version        Print version number

$ incus storage bucket export default bucket0
Backup exported successfully!                      

Archive

The export command creates a tarball from the defined bucket, which is structured as follows:

bucket0.tar.gz
├── backup
│   └── index.yaml
│   └── bucket
    │   └── file1
    │   └── file2

• backup.yml: contains the bucket metadata and keys

name: bucket0
backend: dir
pool: default
config:
  bucket:
    config: {}
    description: ""
    name: bucket0
    s3_url: ""
    location: none
  bucket_keys:
  - description: Admin user
    role: admin
    access-key: OO9O3OXLJWMED0W8P2I5
    secret-key: Ny4W9glDRFks7DzxlPQibREOAUSc1De9Yv0mOkuY
    name: admin
  - description: ""
    role: read-only
    access-key: JIY5PI6RPQWP7CAKJN57
    secret-key: APC95jzb9u7XUKw7J7AWAdVnaNVbjqj5dOLddkhT
    name: reader

• bucket: This folder contains the actual data

API

This includes the following new endpoints (see RESTful API for details):

• GET /1.0/storage-pools/<pool>/buckets/<bucket>/backups

• POST /1.0/storage-pools/<pool>/buckets/<bucket>/backups

• GET /1.0/storage-pools/<pool>/buckets/<bucket>/backups/<name>

• POST /1.0/storage-pools/<pool>/buckets/<bucket>/backups/<name>

• DELETE /1.0/storage-pools/<pool>/buckets/<bucket>/backups/<name>

• GET /1.0/storage-pools/<pool>/buckets/<bucket>/backups/<name>/export

Export

CLI

$ incus storage bucket import
Description:
  Import backups of storage buckets.

Usage:
  incus storage bucket import [<remote>:]<pool> <backup file> [<bucket>] [flags]

Examples:
  incus storage bucket import default backup0.tar.gz
                Create a new storage bucket using backup0.tar.gz as the source.

Flags:
      --target   Cluster member name

Global Flags:
      --debug          Show all debug messages
      --force-local    Force using the local unix socket
  -h, --help           Print help
      --project        Override the source project
  -q, --quiet          Don't show progress information
      --sub-commands   Use with help or --help to view sub-commands
  -v, --verbose        Show all information messages
      --version        Print version number

$ incus storage bucket import default backup.tar.gz bucket0
Backup exported successfully!                      

Design decisions

• One of the existing bucket keys is used by default, as everyone can at least read the bucket.

What's been changed

• The command storage bucket export and storage bucket import has been added to the command line interface (CLI).
• New APIs added
• Added new functionality for performing storage bucket backups and cleaning up expired backups
• Added new functionality for importing storage bucket backups
• Updated API and CLI documentation

[maveonair]

@stgraber I'd love to get some early feedback on the "Storage Bucket Export" implementation as I tackle the import feature.

[stgraber]

By default, we try to use the admin key to export an existing bucket, but it is also possible to specify any key that has access to the bucket with the --key flag.

Given that Incus is in charge of the keys to begin with and currently all keys allow for downloads, I don't think we need to provide that flag, Incus will just pick up a key from the list of keys and use that to fetch the data.

[maveonair]

@stgraber I customized the test suite to run only on the dir and ceph backend as we discussed and the test run was now successful 🎉.

[stgraber]

Looks good. There are a couple of minor commit re-shuffle that I want to do on top of this but I'll do that myself and then merge, thanks!