Proposal: certificate helpers shared by frameworks built on bedrock

[Zaba505]

Summary

Add a nested module, github.com/z5labs/bedrock/pki, with two packages:

• pki decodes PEM into the parts of a tls.Config: a chain and its key for Certificates, and a CertPool for RootCAs or ClientCAs. Each function is a strict version of a standard library function. It fails on input the standard library silently skips, and it fits a config.Value chain.
• pkitest is a certificate authority generated for one test. It is humustest's CA, moved here, with PEM output that pki decodes.

Every framework built on bedrock serves or dials TLS, and the tests of every one of them need certificates. humus has a test CA, and its docs have end users write the decoding themselves, four times over. A command-line framework on bedrock needs the same two things for flags such as --cacert, --cert and --key. bedrock fixes how a certificate is decoded and gives tests a CA. Each framework still builds its own tls.Config and chooses its own TLS policy.

Background

humus leaves decoding to the end user. Its Reading a file says:

a package takes a decoded value, such as a *tls.Config, never a path or a file's contents, so how a certificate is stored and encoded is yours to decide. A further .Parse in the chain does the decoding.

So the end user writes that .Parse, and humus's docs show it four times:

• serverTLS in Serving HTTP, a tls.X509KeyPair(pem, pem) for one PEM file;
• pkcs12TLS in Serving HTTP and serverTLS in Serving gRPC, a PKCS#12 bundle decoded with go-pkcs12;
• clientTLS in Kafka, a client certificate and a CA pool;
• tlsFromPKCS12 in Testing.

The standard library is lenient where bedrock fails loudly. The two functions those helpers call skip what they can't use:

• x509.CertPool.AppendCertsFromPEM skips any block that isn't a certificate, has headers or doesn't parse. It reports only whether it kept at least one. A CA bundle with one corrupt certificate among good ones loads without an error, and the service then distrusts that CA. Kafka's clientTLS checks the bool, so it catches a file with no certificate, but it still misses this case.
• tls.X509KeyPair parses only the leaf. It skips blocks of other types in its certificate input and takes the first private key in its key input. It keeps the rest of the chain unparsed, so a corrupt intermediate is found by the first client that verifies the chain, not at startup.

Neither can fail on a PEM block that is cut off before its END line, because pem.Decode treats it as text. The same happens when two files are joined and the first doesn't end with a newline. -----END CERTIFICATE----------BEGIN PRIVATE KEY----- hides both blocks, so slices.Concat(certPEM, keyPEM) can decode to nothing at all.

humus also has a test CA. humustest.CA generates a CA for each test and issues certificates for names. It hands them out as a tls.Certificate, a CertPool or a PKCS#12 bundle. humus's tests and docs use it in over a hundred places, and its Testing guide gives the reason: a test never uses a fixed key, certificate or password, so no secret is written into the repository and nothing is shared between tests.

A second framework needs both. A command-line framework on bedrock is planned (see #614). It dials more than it serves. Its users pass a CA, a certificate and a key the way they pass them to curl and docker, or inline as in a kubeconfig's certificate-authority-data, and its tests need a CA as humus's do. Today it has two bad options:

• Import humustest, which links in humus, the OpenTelemetry SDK and grpc-go (see Proposal: a Host shared by frameworks built on bedrock #614).
• Copy the CA and the decoders. Copies drift, and what drifts here is what a certificate file may hold and which mistakes fail at startup.

Left open by #614. Its settled question 3 kept humustest's public CA where it is, and said a bedrock package of TLS, CA and certificate helpers may deserve a proposal of its own. This is that proposal.

Proposed API

package pki // github.com/z5labs/bedrock/pki

// Certificates decodes the certificates in pemData, in order. pemData is a
// series of PEM blocks. Text between blocks is ignored, as in a CA bundle
// with comments. Certificates fails on data that holds no block, on a block
// that isn't a CERTIFICATE, has headers or doesn't parse, and on a block
// with no END line. Its errors name a block by its position, counting
// from 1, and wrap x509's parse error.
func Certificates(pemData []byte) ([]*x509.Certificate, error)

// Pool returns a CertPool that holds the certificates in pemData, decoded
// as by Certificates, for a tls.Config's RootCAs or ClientCAs. Where
// x509.CertPool.AppendCertsFromPEM skips a block it can't use, Pool fails.
func Pool(pemData []byte) (*x509.CertPool, error)

// KeyPair decodes certPEM's certificate chain, leaf first, and keyPEM's
// private key into a tls.Certificate for a tls.Config's Certificates, with
// Leaf set. Like tls.X509KeyPair, it takes a file that holds both as both
// arguments, so certPEM may hold the key and keyPEM may hold the chain.
//
// It accepts what X509KeyPair accepts and fails where X509KeyPair would
// skip or defer: on a block in either input that is neither a CERTIFICATE
// nor a private key, on a certificate of the chain that doesn't parse, on
// a second private key in keyPEM, and on a block with no END line. A key
// that doesn't match the leaf fails, as in X509KeyPair, and so does a key
// encrypted with a password.
func KeyPair(certPEM, keyPEM []byte) (tls.Certificate, error)

package pkitest // github.com/z5labs/bedrock/pki/pkitest

// CA is a certificate authority generated for one test, with a new ECDSA
// P-256 key. It is valid from an hour before time.Now() to a day after.
// Inside a testing/synctest bubble that is the bubble's clock, so build a
// CA inside the bubble that uses it.
type CA struct{ /* unexported fields */ }

// NewCA generates a CA. It fails the test through tb if generation fails.
func NewCA(tb testing.TB) *CA

// Pool returns a CertPool holding only the CA's certificate, for a
// tls.Config's RootCAs or ClientCAs.
func (ca *CA) Pool() *x509.CertPool

// PEM returns the CA's certificate as a PEM block, which pki.Pool decodes.
func (ca *CA) PEM() []byte

// Issue returns a certificate signed by the CA, with a new key, for names,
// which are DNS names or IP literals. It is valid for server and client
// authentication, from an hour before time.Now() to a day after. A client
// certificate needs no names.
func (ca *CA) Issue(tb testing.TB, names ...string) *Certificate

// Certificate is a certificate issued by a CA, with its private key.
type Certificate struct{ /* unexported fields */ }

// TLS returns the certificate, its chain and its key, for a tls.Config's
// Certificates.
func (c *Certificate) TLS() tls.Certificate

// PEM returns the chain, the certificate then the CA's, and the private
// key, each as PEM, which pki.KeyPair decodes: pki.KeyPair(c.PEM()).
func (c *Certificate) PEM() (certPEM, keyPEM []byte)

Here a service reads its certificate and key from the two files that a Kubernetes TLS secret mounts, and requires each client to present a certificate signed by a CA in a third file. It reads the files through the filesystem of its Host (#614), so a test reads them from its own directory. A missing variable or file fails as a config error named after its variable. A pair that doesn't decode fails under TLS_KEY_FILE, the variable that completes it. config.Struct reports every failure in one error:

// FS is what serverTLS reads files through: host's OSFS in production and
// hosttest's FS in a test.
type FS interface {
	ReadFile(name string) ([]byte, error)
}

func serverTLS(ctx context.Context, env bedrock.Environment, fsys FS) (*tls.Config, error) {
	file := func(name string) config.Value[[]byte] {
		return config.Env(env, name).Parse(fsys.ReadFile)
	}
	pair := file("TLS_CERT_FILE").FlatMap(func(certPEM []byte) config.Value[tls.Certificate] {
		return file("TLS_KEY_FILE").Parse(func(keyPEM []byte) (tls.Certificate, error) {
			return pki.KeyPair(certPEM, keyPEM)
		})
	})
	return config.Struct(func(r *config.Resolver) *tls.Config {
		return &tls.Config{
			Certificates: []tls.Certificate{r.Get(pair)},
			ClientCAs:    r.Get(file("TLS_CLIENT_CA_FILE").Parse(pki.Pool)),
			ClientAuth:   tls.RequireAndVerifyClientCert,
		}
	}).Read(ctx)
}

Its test writes a new CA's files into hosttest's directory and runs both handshakes inside a synctest bubble, with no real socket and no certificate in the repository:

func TestServerTLS_requires_a_client_certificate(t *testing.T) {
	synctest.Test(t, func(t *testing.T) {
		ca := pkitest.NewCA(t)
		certPEM, keyPEM := ca.Issue(t, "api.internal").PEM()
		h := hosttest.New(t, hosttest.WithFiles(map[string][]byte{
			"tls/tls.crt":       certPEM,
			"tls/tls.key":       keyPEM,
			"tls/client-ca.crt": ca.PEM(),
		}))
		env := bedrock.NewStaticEnvironment(map[string]string{
			"TLS_CERT_FILE":      "tls/tls.crt",
			"TLS_KEY_FILE":       "tls/tls.key",
			"TLS_CLIENT_CA_FILE": "tls/client-ca.crt",
		})
		cfg, err := serverTLS(t.Context(), env, h.FS())
		require.NoError(t, err)

		// handshake runs a TLS handshake over a net.Pipe and returns the
		// server's error.
		handshake := func(client *tls.Config) error {
			c, s := net.Pipe()
			errs := make(chan error, 1)
			go func() {
				errs <- tls.Server(s, cfg).HandshakeContext(t.Context())
				_ = s.Close()
			}()
			_ = tls.Client(c, client).HandshakeContext(t.Context())
			_ = c.Close()
			return <-errs
		}

		client := &tls.Config{RootCAs: ca.Pool(), ServerName: "api.internal"}
		require.Error(t, handshake(client))

		client.Certificates = []tls.Certificate{ca.Issue(t, "billing").TLS()}
		require.NoError(t, handshake(client))
	})
}

Both blocks compile against stubs of this API and of hosttest's. The handshake helper, written with the standard library in place of pki and pkitest, passes under -race. Without a client certificate the server fails with tls: client didn't provide a certificate.

Design decisions

• A nested module, not the root module. It follows fit, circuit, retry and host, so the root module stays exactly the process model. Its library code imports only the standard library, and it requires no other bedrock module, not even for tests. A CLI framework that wants certificates gets nothing else. For the same reason, host's tests keep the unexported CA that story(host): hosttest's network, ported from humustest #616 gives them.
• Bytes in, decoded values out. pki never opens a file. A caller reads the file through its framework's filesystem, such as host.OSFS.ReadFile, so a test reads from its own directory. Each decoder then runs as a step of a config.Value chain, where its error is named after the variable, the way humus's Reading a file already asks. PEM that arrives inline, from a variable or a kubeconfig, decodes the same way.
• Strict versions of standard library functions, with no new format. pki accepts a subset of what the standard library accepts. A valid input gives the same result, so switching to pki changes only which mistakes fail at startup. Text between blocks stays allowed, because real bundles have it: Mozilla's CA bundle has comment headers, and openssl s_client -showcerts prints subject= lines.
• KeyPair has the shape of tls.X509KeyPair. Most deployments keep the certificate and the key in two files. Examples are a Kubernetes TLS secret's tls.crt and tls.key, cert-manager's output, docker's --tlscert and --tlskey, curl's --cert and --key, and OTLP's OTEL_EXPORTER_OTLP_CLIENT_CERTIFICATE and _CLIENT_KEY. Two arguments avoid joining the files, which pem.Decode can misread (see Background), and a file that holds both is passed as both, as with X509KeyPair today. Certificates and Pool take one argument, because a CA bundle is one file, and so they drop straight into .Parse.
• KeyPair doesn't verify the chain, its order or its dates. A server may send a partial chain on purpose, and whether a chain is trusted is the client's question. Failing on a certificate that has expired or is about to is a policy, and a framework or an App adds it as a .Validate step in its own chain.
• No *tls.Config constructors. Once its parts are decoded, a config is a struct literal, as in the example above. Every field beyond the parts is a choice that humus and the CLI framework make differently: MinVersion, ClientAuth, NextProtos and CipherSuites. humus's CheckTLS already encodes what HTTP/2 requires. bedrock fixes how the parts are decoded, and the frameworks choose the config. See open question 2.
• PEM only. The standard library has no PKCS#12 decoder, and pki imports only the standard library. PEM is what Kubernetes, cert-manager, Vault and SPIFFE hand a process. See open question 4.
• System roots are an ambient leak, and pki doesn't wrap them. A tls.Config whose RootCAs is nil, and x509.SystemCertPool, read SSL_CERT_FILE and SSL_CERT_DIR from the real environment and the system's certificate files from the real filesystem. Since Go 1.27, that also replaces the platform verifier on macOS and Windows when either variable is set. Reading them through Process would mean reimplementing Go's root discovery, and macOS and Windows verify through platform APIs that a file can't stand in for. So pki's package doc and bedrock's Ambient leaks list say so, and a test sets RootCAs from its CA.
• pkitest is humustest's CA, moved, with these changes:
  • PEM output. CA.PEM and Certificate.PEM are what a test writes into the App's filesystem, and pki decodes them, so the two packages are each other's tests. Certificate.PEM returns the cert and the key apart, for the two-file layout. A test that wants one file writes slices.Concat(certPEM, keyPEM), which is safe here because pem.EncodeToMemory ends every block with a newline.
  • No PKCS#12. Encoding it needs go-pkcs12. humustest keeps its PKCS12 method, built on pkitest's TLS().
  • No hand-rolled serial number. x509.CreateCertificate generates a serial number that follows RFC 5280 when the template has none, so humustest's loop over rand.Int goes.
  • No options. Validity is fixed. A test of an expired certificate sleeps past it inside a synctest bubble. In a prototype, a handshake a bubble-day later failed with x509.Expired after 0.01s of real time. A wrong name is an Issue for another name, and an untrusted CA is a second NewCA.
• Names don't repeat the package. That gives pki.Certificates and pkitest.Certificate, where certs would give certs.Certificates. See open question 1.

Testing

• Coverage. Statement coverage of pki and pkitest is 100.0%, and the coverage gate covers the new module.
• Equivalence with the standard library. A table of valid inputs checks that pki returns what the standard library returns: KeyPair against tls.X509KeyPair, and Pool against AppendCertsFromPEM, compared with CertPool.Equal. The table covers RSA, ECDSA and Ed25519 keys, PKCS#1, PKCS#8 and SEC 1 key blocks, one file passed as both arguments, and bundles with text between blocks.
• Strictness. A second table holds every input the standard library accepts leniently and pki rejects: a stray block type, a corrupt intermediate, a second key, a block with no END line, two files joined without a newline, and a certificate with headers. Each row checks that the standard library accepts the input, or silently drops part of it, and that pki fails.
• A fuzz test checks that pki never accepts what the standard library rejects, and that every input pki accepts gives the standard library's result. Its seed corpus runs in the default go test.
• Round trip. Every pkitest output decodes with pki and completes a TLS handshake over net.Pipe inside a synctest bubble, both server-only and mutual.

Migration from today

• bedrock. The change is additive.
  • A new nested module. It needs CI build, vet, test and lint steps, the coverage gate, a depguard rule, and the CLAUDE.md commands and package layout. tag-modules.sh already finds every nested go.mod.
  • "Ambient leaks" in the root package doc gains the system roots, as above.
  • README gains a ## pki section.
• humus. humus decides through its own proposal. Nothing in this proposal changes humus. That proposal decides whether to:
  • replace the decoders in its docs with pki;
  • build humustest.CA on pkitest, or replace it, keeping PKCS12 in humustest.
• The CLI framework decodes --cacert, --cert and --key with pki, and its tests use pkitest.

Open questions

1. The name. I recommend pki and pkitest. Neither repeats in its names, and neither collides with a standard library package that users import next to it. crypto/x509/pkix is close, but only code that builds certificates imports it, and pkitest builds them for its users. The alternatives:

   • certs, which repeats in certs.Certificates;
   • tlsconfig, which suggests that the package builds a *tls.Config.

2. Should bedrock build a *tls.Config? Examples would be pki.ServerConfig(pair) or pki.ClientConfig(roots, pairs...). I recommend no, as argued in Design decisions. The decoders are the part every framework writes the same way, and the config is the part they write differently.

3. Certificate rotation. A service whose certificate rotates on disk serves it with GetCertificate. humus's docs mention that, and leave writing it to the end user. A reloader has its own decisions:

   • when it re-reads, on a schedule or at a handshake once the certificate is old enough;
   • what it does when a re-read fails partway through a rotation;
   • how it reports a failure.

   A CLI rarely needs one, because its process is short-lived. I recommend a follow-up proposal, built on KeyPair, once this one is settled.

4. PKCS#12. I recommend leaving it out. Decoding it needs a third-party module, software.sslmate.com/src/go-pkcs12, and only fit has an exception to the standard-library rule. PKCS#12 is mostly a Java keystore format, and humus's docs already show it as the end user's choice with go-pkcs12. Adding it later is additive.

Out of scope

• Certificate rotation, pending open question 3.
• Issuing certificates in production, such as an ACME client or a CA service. pkitest's CA is for tests.
• PKCS#12, encrypted keys, and Java keystores.
• Reading system roots through Process; see Design decisions.
• Revocation, that is, CRLs and OCSP.
• TLS policy: minimum versions, cipher suites, ALPN and client authentication. These belong to each framework, as humus's CheckTLS does.
• Intermediate CAs in pkitest. Adding them later is additive, if a test needs a chain longer than the leaf and the root.