-
Notifications
You must be signed in to change notification settings - Fork 1.4k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add design doc for Custom CA support for S3 BSLs and Velero Installat…
…ion (#2259) * Add design doc for Custom CA support for S3 Signed-off-by: Dylan Murray <dymurray@redhat.com>
- Loading branch information
Showing
1 changed file
with
132 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,132 @@ | ||
# Custom CA Bundle Support for S3 Object Storage | ||
|
||
It is desired that Velero performs SSL verification on the Object Storage | ||
endpoint (BackupStorageLocation), but it is not guaranteed that the Velero | ||
container has the endpoints' CA bundle in it's system store. Velero needs to | ||
support the ability for a user to specify custom CA bundles at installation | ||
time and Velero needs to support a mechanism in the BackupStorageLocation | ||
Custom Resource to allow a user to specify a custom CA bundle. This mechanism | ||
needs to also allow Restic to access and use this custom CA bundle. | ||
|
||
## Goals | ||
|
||
- Enable Velero to be configured with a custom CA bundle at installation | ||
- Enable Velero support for custom CA bundles with S3 API BackupStorageLocations | ||
- Enable Restic to use the custom CA bundles whether it is configured at installation time or on the BackupStorageLocation | ||
- Enable Velero client to take a CA bundle as an argument | ||
|
||
## Non Goals | ||
|
||
- Support non-S3 providers | ||
|
||
## Background | ||
|
||
Currently, in order for Velero to perform SSL verification of the object | ||
storage endpoint the user must manually set the `AWS_CA_BUNDLE` environment | ||
variable on the Velero deployment. If the user is using Restic, the user has to | ||
either: | ||
1. Add the certs to the Restic container's system store | ||
1. Modify Velero to pass in the certs as a CLI parameter to Restic - requiring | ||
a custom Velero deployment | ||
|
||
## High-Level Design | ||
|
||
There are really 2 methods of using Velero with custom certificates: | ||
1. Including a custom certificate at Velero installation | ||
1. Specifying a custom certificate to be used with a `BackupStorageLocation` | ||
|
||
### Specifying a custom cert at installation | ||
|
||
On the Velero deployment at install time, we can set the AWS environment variable | ||
`AWS_CA_BUNDLE` which will allow Velero to communicate over https with the | ||
proper certs when communicating with the S3 bucket. This means we will add the | ||
ability to specify a custom CA bundle at installation time. For more | ||
information, see "Install Command Changes". | ||
|
||
On the Restic daemonset, we will want to also mount this secret at a pre-defined | ||
location. In the `restic` pkg, the command to invoke restic will need to be | ||
updated to pass the path to the cert file that is mounted if it is specified in | ||
the config. | ||
|
||
This is good, but doesn't allow us to specify different certs when | ||
`BackupStorageLocation` resources are created. | ||
|
||
### Specifying a custom cert on BSL | ||
|
||
In order to support custom certs for object storage, Velero will add an | ||
additional field to the `BackupStorageLocation`'s provider `Config` resource to | ||
provide a secretRef which will contain the coordinates to a secret containing | ||
the relevant cert file for object storage. | ||
|
||
In order for Restic to be able to consume and use this cert, Velero will need | ||
the ability to write the CA bundle somewhere in memory for the Restic pod to | ||
consume it. | ||
|
||
To accomplish this, we can look at the code for managing restic repository | ||
credentials. The way this works today is that the key is stored in a secret in | ||
the Velero namespace, and each time Velero executes a restic command, the | ||
contents of the secret are read and written out to a temp file. The path to | ||
this file is then passed to restic and removed afterwards. pass the path of the | ||
temp file to restic, and then remove the temp file afterwards. See ref #1 and #2. | ||
|
||
This same approach can be taken for CA bundles. The bundle can be stored in a | ||
secret which is referenced on the BSL and written to a temp file prior to | ||
invoking Restic. | ||
|
||
[1](https://github.com/vmware-tanzu/velero/blob/master/pkg/restic/repository_manager.go#L238-L245) | ||
[2](https://github.com/vmware-tanzu/velero/blob/master/pkg/restic/common.go#L168-L203) | ||
|
||
## Detailed Design | ||
|
||
The `AWS_CA_BUNDLE` environment variable works for the Velero deployment | ||
because this environment variable is passed into the AWS SDK which is used in | ||
the [plugin][1] to build up the config object. This means that a user can | ||
simply define the CA bundle in the deployment as an env var. This can be | ||
utilized for the installation of Velero with a custom cert by simply setting | ||
this env var to the contents of the CA bundle, or the env var can be mapped to | ||
a secret which is controlled at installation time. I recommend using a secret | ||
as it makes the Restic integration easier as well. | ||
|
||
At installation time, if a user has specified a custom cert then the Restic | ||
daemonset should be updated to include the secret mounted at a predefined path. | ||
We could optionally use the system store for all custom certs added at | ||
installation time. Restic supports using the custom certs [in addition][3] to | ||
the root certs. | ||
|
||
In the case of the BSL being created with a secret reference, then at runtime | ||
the secret will need to be consumed. This secret will be read and applied to | ||
the AWS `session` object. The `getSession()` function will need to be updated | ||
to take in the custom CA bundle so it can be passed [here][4]. | ||
|
||
The Restic controller will need to be updated to write the contents of the CA | ||
bundle secret out to a temporary file inside of the restic pod.The restic | ||
[command invocation][2] will need to be updated to include the path to the file | ||
as an argument to the restic server using `--cacert`. For the path when a user | ||
defines a custom cert on the BSL, Velero will be responsible for updating the | ||
daemonset to include the secret mounted as a volume at a predefined path. | ||
|
||
Where we mount the secret is a fine detail, but I recommend mounting the certs | ||
to `/certs` to keep it in line with the other volume mount paths being used. | ||
|
||
### Install command changes | ||
|
||
The installation flags should be updated to include the ability to pass in a | ||
cert file. Then the install command would do the heavy lifting of creating a | ||
secret and updating the proper fields on the deployment and daemonset to mount | ||
the secret at a well defined path. | ||
|
||
### Velero client changes | ||
|
||
Since the Velero client is responsible for gathering logs and information about | ||
the Object Storage, this implementation should include a new flag `--cacert` | ||
which can be used when communicating with the Object Storage. Additionally, the | ||
user should be able to set this in their client configuration. The command | ||
would look like: | ||
``` | ||
$ velero client config set cacert PATH | ||
``` | ||
|
||
[1]: https://github.com/vmware-tanzu/velero-plugin-for-aws/blob/master/velero-plugin-for-aws/object_store.go#L135 | ||
[2]: https://github.com/vmware-tanzu/velero/blob/master/pkg/restic/command.go#L47 | ||
[3]: https://github.com/restic/restic/blob/master/internal/backend/http_transport.go#L81 | ||
[4]: https://github.com/vmware-tanzu/velero-plugin-for-aws/blob/master/velero-plugin-for-aws/object_store.go#L154 |