Certificate refresh interval doesn't seem to work

[jmmv]

I'm currently using bb-clientd with some short-lived certificates and it seems as if the logic to refresh them isn't working correctly. So far, I've worked around this by making bb-clientd restart whenever the certificate files change, but it would be nice to have the built-in refresh support work properly.

I'm configuring bb-clientd with something like this:

tls: {
  serverCertificateAuthorities: std.extVar('BUILDBARN_PEM_CONTENT'),
  clientKeyPair: {
    files: {
      certificatePath: std.extVar('CERTIFICATE_PATH'),
      privateKeyPath: std.extVar('PRIVATE_KEY_PATH'),
      refreshInterval: '60s',
    },
  },
}

And I run Bazel builds with these flags (which means I'm using your patch to use bb-clientd for the output tree):

• --remote_executor=unix:/home/jmmv/.cache/bb_clientd/grpc
• --remote_instance_name=prod/experiment
• --remote_output_service=unix:/home/jmmv/.cache/bb_clientd/grpc
• --remote_output_service_output_path_prefix=/home/jmmv/bb_clientd/outputs
• --tls_certificate=/path/to/file
• --tls_client_certificate=/path/to/file
• --tls_client_key=/path/to/file

The TLS settings given to Bazel match exactly those given to bb-clientd above.

Here is the sequence of events to reproduce the issue:

1. Get new certificates.
2. Start bb-clientd.
3. Run a Bazel build and see it succeed.
4. Wait for certificate expiration.
5. Run the build again, which fails with:
   ERROR: Failed to query remote execution capabilities: UNAUTHENTICATED: Cannot validate TLS client certificate: x509: certificate has expired or is not yet valid: current time 2023-02-13T17:11:02Z is after 2023-02-13T17:06:10Z
6. Refresh certificate.
7. Run the build again right after, which fails with:
   ERROR: Failed to query remote execution capabilities: UNAUTHENTICATED: Cannot validate TLS client certificate: x509: certificate has expired or is not yet valid: current time 2023-02-13T17:11:02Z is after 2023-02-13T17:06:10Z
8. Wait for the refresh interval configured in bb-clientd:
9. Run the build again, which fails with a different message:
   ERROR: Failed to query remote execution capabilities: UNAUTHENTICATED: Backend "": Backend "prod": Cannot validate TLS client certificate: x509: certificate has expired or is not yet valid: current time 2023-02-13T17:11:22Z is after 2023-02-13T17:06:10Z
10. At this point there is no way to get the build to work.
11. Restart bb-clientd (NOT Bazel).
12. Build succeeds again.

[joeljeske]

Hey @jmmv good callout, currently gRPC Client connections have broken certificate rotation. I just documented the issue here, buildbarn/bb-storage#162. I am actively tracking this issue, and would like to resolve the issue once we gain proper support in grpc-go.