Add design doc for Custom CA support for S3 BSLs and Velero Installation #2259
Conversation
Signed-off-by: Dylan Murray <dymurray@redhat.com>
design/custom-ca-support.md
Outdated
| additional field to the `BackupStorageLocation`'s provider `Config` resource to | ||
| provide a secretRef which will contain the coordinates to a secret containing | ||
| the relevant cert file for object storage. Then, in order for Restic to be able | ||
| to consume and use this cert, Velero will need to modify the Restic daemonset |
There was a problem hiding this comment.
Is this an accepted path for Velero? It feels odd to me to update the daemonset for Restic on the fly when creating BSLs as this could interrupt some activity Restic is undertaking.
However, without modifying the restic daemonset I don't see a feasible way to get the custom CA bundle into Restic at runtime. In our PoC we had to use an initcontainer on Restic which would symlink a secret that is mounted into the Restic container. This would allow us to modify a secret from Velero and have the associated changes populated in the Restic container. I do not think that's a viable approach moving forward though.
There was a problem hiding this comment.
One idea: take a look at Velero's code for managing restic repo credentials, i.e. the encryption key. The way this works today is that the key is stored in a secret in the velero namespace, and each time we exec a restic command, we read the contents of the secret, write it out to a temp file, pass the path of the temp file to restic, and then remove the temp file afterwards. This has worked pretty well in practice*.
We might be able to take a similar approach for the CA bundles, where they're stored in secrets, the BSL has a reference to them, and they're projected into a temp file just-in-time for restic invocations.
Here are a few links to relevant sections of the current code:
https://github.com/vmware-tanzu/velero/blob/master/pkg/restic/repository_manager.go#L238-L245
https://github.com/vmware-tanzu/velero/blob/master/pkg/restic/common.go#L168-L203
- as an aside, yes, this is over-complicated given that we're currently using a static/non-secret encryption key for restic repos, but this was implemented before we decided to go with that approach instead of using unique, secret keys per repo.
There was a problem hiding this comment.
@skriss thanks a lot for the links I definitely think we could follow a similar pattern for the CA bundle here since restic allows for a --cacert command flag which we can specify like we do the password file here: https://github.com/vmware-tanzu/velero/blob/master/pkg/restic/command.go#L86.
as an aside, yes, this is over-complicated given that we're currently using a static/non-secret encryption key for restic repos
Makes sense. I have faced similar problems like this in the past and it seems like this is a common pattern used to solve the issue of needing data stored somewhere in memory of a pod that a controller will manage.
I will try something like this out and update the document.
| Custom Resource to allow a user to specify a custom CA bundle. This mechanism | ||
| needs to also allow Restic to access and use this custom CA bundle. | ||
|
|
||
| ## Goals |
There was a problem hiding this comment.
Does this support adding the certs on the client side? I know velero client-side commands such as velero (backup|restore) (describe|logs) on AWS/GCP/Azure get a 1-time use URL for retrieving files from object store, and on Minio they use a BSL's Spec.Config.publicUrl value in order to get non-encrypted values.
There was a problem hiding this comment.
It currently does not but I wouldn't rule that out. Are you thinking that the velero install command would install the cert on the client's machine? Or that the velero client commands would have a new flag --cacert which a user can optionally specify?
There was a problem hiding this comment.
I'm not exactly sure, honestly. I think either would work, so long as it's documented. A --cacert flag would probably be easier since certs can be managed in several different ways depending on client OS.
Mostly, I'd like to avoid users hitting issues like this one often. #2256
dymurray
changed the title
|
@dymurray just checking in to see if you were able to try out the restic approach per #2259 (comment), and if you need any other input! |
|
@skriss I was on PTO for the better part of last week. Have begun putting together the PoC for this this morning. Hoping to have it confirmed by the community meeting tomorrow and can update the design doc. |
|
Can confirm that the approach of using the methods shown in the |
Update design to write CA bundle to a temp file Include information about client flag Signed-off-by: Dylan Murray <dymurray@redhat.com>
design/custom-ca-support.md
Outdated
| Since the Velero client is responsible for gathering logs and information about | ||
| the Object Storage, this implementation should include a new flag `--cacert` | ||
| which can be used when communicating with the Object Storage. Additionally, the | ||
| user should be able to set this in their client configuration. |
There was a problem hiding this comment.
To put more detail on it, I would probably put it under velero client config set cacert PATH if someone would like to permanently set the value.
There was a problem hiding this comment.
Thx will add this as a note
|
|
||
| ### Install command changes | ||
|
|
||
| The installation flags should be updated to include the ability to pass in a |
There was a problem hiding this comment.
@carlisia This is relevant to #2202. My guess would probably be part of the plugin add command there, with a new option for certs that would create the secret and do the updating on the deployment, and restic DaemonSet. I think this is only valid for ObjectStore plugins, at least the restic DaemonSet part. I'm envisioning the command would take a path to the cert file, create the Secret in the cluster, then put a secretRef in the Velero Deployment/restic DaemonSet.
We'd also need to have the backup-location (set|create) commands have an argument to take a reference to a secret name (which is the cert) in the same namespace. Possibly the snapshot-location (set|create) commands, if it's relevant to them. The command would create or edit the BSL (or VSL) to have a reference to the Secret within the same namespace.
There was a problem hiding this comment.
Copy that! I'll add it to the doc, thank you.
There was a problem hiding this comment.
Seems it will work the same way as with the credentials for the provider (as far as adding it to the cluster).
There was a problem hiding this comment.
Yep, I think it'll work the same way, it's just a Secret with different contents.
Signed-off-by: Dylan Murray <dymurray@redhat.com>
|
Note that on Linux it appears the restic cacert can be set using golang standard pki environment variables as described here: https://go-review.googlesource.com/c/go/+/36093/. In particular env var SSL_CERT_FILE is known to work in the daemonset. |
Signed-off-by: Dylan Murray dymurray@redhat.com
Design for #1027