Manage secrets with sealed secrets

[jjnesbitt]

Currently, secrets are managed in an ad-hoc way, where the secret is created in the cluster, and then added here in a completely commented file named secrets-dummy, purely for reference. Sealed Secrets provide a way for us to securely commit our secrets into our codebase, without the need for any workarounds.

This PR consists of the following:

1. The necessary helm repositories/charts for sealed secrets
2. SealedSecret resource defintions for all of the secrets which had a secrets-dummy file committed. These contain the encrypted data pulled from the live cluster
3. Scripts to help facilitate the management of secrets
4. Documentation detailing our use of sealed secrets

For all of the secrets encoded in the sealed-secrets.yaml files included in this PR, I've added the "sealedsecrets.bitnami.com/managed" = "true" annotation, which will allow this change to take effect correctly.

[jjnesbitt]

Waiting to merge this until #404 is merged. I'll then rebase this off of that and deal with conflicts, as well as configure staging, since staging isn't currently covered by this PR.

UPDATE: This has been completed.

[mvandenburgh]

Looks good to me, although i haven't tested it.

[mvandenburgh]

Should we disable flux in production, merge this, and re-enable it when we're sure it works ok in staging? I'm not sure the best way to test all of this tbh, unless you've already done so.

[jjnesbitt]

Should we disable flux in production, merge this, and re-enable it when we're sure it works ok in staging? I'm not sure the best way to test all of this tbh, unless you've already done so.

Just to clarify, I have already tested this to the best of my ability by doing the following:

1. Applied one or more of these files (under a different name) to the cluster, checking the contents of the unsealed secret and verifying that it's correct
2. Verified that the private and public key pair match by temporarily storing the private key locally and using the kubeseal recovery mode to decrypt the secret files locally, verifying that their contents are what they should be
3. Applied the "sealedsecrets.bitnami.com/managed" = "true" annotation to all existing secrets that have corresponding sealed-secrets.yaml files in this PR
4. Verified that an existing secret which has the above annotation can be correctly "taken control of" by the sealed secrets manager (i.e. that the sealed secret would be unsealed and the regular secret data updated without issue, even though all sealed data in this PR is the same as the existing secrets).

I'm still happy to do a test in staging first, but I don't think it'll really be testing what we care about, since the staging cluster uses a different key pair, and doesn't contain most of the secrets we're encoding here. Essentially what I'm saying is, I think the most meaningful test is verifying that we've sealed all of the existing secret data correctly, which I've done to the best of my knowledge.

[mvandenburgh]

I'm still happy to do a test in staging first, but I don't think it'll really be testing what we care about, since the staging cluster uses a different key pair, and doesn't contain most of the secrets we're encoding here. Essentially what I'm saying is, I think the most meaningful test is verifying that we've sealed all of the existing secret data correctly, which I've done to the best of my knowledge.

Agreed, sounds like you've tested this reasonably so I think it's good to merge 👍