engineccl: Support JWKS format for encryption-at-rest keys

[bdarnell]

Is your feature request related to a problem? Please describe.

The enterprise encryption flag contains support for key rotation, but not for migrating to new encryption algorithms. We need to introduce new configurability to select an encryption algorithm. This could be done as a part of the command line flag, but it would be even better to attach algorithm information to the key files themselves to take advantage of the ability to reload keys without restarting the server.

Describe the solution you'd like

Replace our current non-standard key file format with the JWKS format (specified in RFC 7517 and already used elsewhere in CRDB for SSO protocols). The key file must be a JWKS object that contains exactly one key, while the old-key file could optionally gain the ability to contain multiple keys (to facilitate frequent key rotations without waiting for data to be rewritten). (Is there a better or more JWKS-standard way to designate one active key out of a key set?)

Our current key files are a concatenation of a binary key id and the raw key data. The new key files would look like

{
    "keys": [
        {
            "kty": "oct",  // "octets", indicates that this is a single blob as opposed to e.g. a pub/priv pair
            "kid": "base64'd key id",
            "k": "base64'd key data",
            "alg": "cockroach-aes-ctr-v1",
        }
    ],
}

The point of the new format is the "alg" field, where we would define identifiers for our encryption-at-rest algorithms. Unfortunately we'd have to define our own algorithm IDs, since none of the standardized ones are suitable for our purpose (they are all designed for messages small enough to be decrypted all at once, and not for large files like our sstables). While cockroach-aes-ctr-v1 is something that should remain cockroach-specific (aes-ctr-v2 is a little better but still not AEAD), we could consider in the future advocating for standardization of a better seekable algorithm/format (i.e. one that divides the data stream into equal-sized chunks and applies AES-GCM on a per-chunk basis).

Concretely, this means the following changes:

• cockroach gen encryption-key gains the option to generate the new format
• The key loading code learns to recognize the new format. Unfortunately the old format doesn't contain any version marker, but since it contains purely random data it's safe to assume that any key file that parses as json is in the new format.
• The alg field is used to select between encryption implementations, and is persisted in the per-file EncryptionSettings in the registry.

Describe alternatives you've considered

The --enterprise-encryption flag could gain a new alg argument and leave the key files unchanged. But this means that changing algorithms requires a node restart, and we'd probably have to build some more observability tools to track the progress of the change.

The JWKS format in theory makes it possible to use off-the-shelf tools instead of writing our own, but the need to use custom alg values limits the applicability of existing tools. If the JWKS format had notable drawbacks this might be a reason to design our own, but I don't see any reason to do so.

Other key file formats are in use in the public-key crypto world (PEM, etc), but as far as I can tell there's not much besides JWKS that is widely used for symmetric keys.

Integrating encryption into the pebble WAL and SST format at a higher level would be a better way to use AEAD algorithms. If we went that route the existing encryption at rest code could become obsolete.

Additional context

This is the replacement for #118824

Open question: Is the "alg" aes-ctr and it supports multiple key sizes, or are there separate algs for each key size? The standard JWKS algorithms include the key size in the alg name, and I have a slight preference for following that pattern going forward.

CCing some folks who have reviewed previous PRs in this series: @RaduBerinde @sumeerbhola @rsevinsky-cr

Jira issue: CRDB-36304