Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[TLS] Support for hot swapping of node certificates #11531

Closed
streddy-yb opened this issue Feb 18, 2022 · 3 comments
Closed

[TLS] Support for hot swapping of node certificates #11531

streddy-yb opened this issue Feb 18, 2022 · 3 comments
Assignees
@bmatican @es1024
Labels
area/docdb YugabyteDB core features

Comments

@streddy-yb
Copy link
Contributor

Currently the following use cases require a rolling restart which is not ideal from operational perspective as rotating certs more frequently (ex: every 3-months) is becoming a standard practice.

  • Rotating node certs: Node certs are about to expire and user needs to rotate the node certs. Root cert itself is valid.
  • Rotating root & node certs: If root cert is expiring, then ability to rotate both hot swap root cert and node cert is required
  • Rotating certs for cdc (i.e, certs placed for certs_for_cdc_dir): Source universe cert changes required updating these certs.

The ask is to support hot swapping certificates for the above scenarios without requiring a rolling restart. Expose a mechanism to trigger the hot swap via appropriate interfaces (yb-admin, RPC, etc).

There is a separate request to support multiple CA cert specifications for master/tserver (#5730) which is not related to this but that would also help managing multiple certs when integrating with 3rd party systems like ldap.
@streddy-yb streddy-yb added the area/docdb YugabyteDB core features label Feb 18, 2022
@bmatican bmatican self-assigned this Feb 28, 2022
@spolitov
Copy link
Contributor

We have the following related files: ent/src/yb/server/secure.cc/h and yb/rpc/secure_stream.cc/h
The easier way to rotate certificate and private key would be adding interface to SecureContext to update them.

@rthallamko3
Copy link
Contributor

@es1024 , Can you get some context on this.

@bmatican bmatican mentioned this issue Apr 5, 2022
Closed
es1024 added a commit that referenced this issue Apr 19, 2022
@es1024
[#11531] docdb: Added hot swapping of server TLS certificates
73a3ca4 
Summary:
Added `yb-ts-cli reload_certificates` to make servers reload the CA certificates, node
certificate, and node private key from disk once again.

Also regenerated test certificates to have endDate=99991231235959Z instead of various
dates in the next couple of years.

This change should allow for node certificate/key rotation without restart for node to node encryption
by just changing the appropriate crt/key files on the node and triggering a reload. CA certificate rotation can be
safely executed in a three step process:

1. Append new CA certificate to the end of ca.cert and trigger a reload on all nodes. This CA must
    either be signed with the same private key as the original CA or have a different common name.
2. Changes node certificates/keys on each node. All nodes at this point can accept certificates
    signed by both the old and the new CA, so there are no issues.
3. Remove old CA certificate and trigger a reload on all nodes.

Reloading CQLServer/postgres certificates (required for when client/server encryption is enabled) will
be handled in a separate diff.

Test Plan: `ybd --gtest_filter ExternalMiniClusterSecureTest.ReloadCertificates`

Reviewers: sergei, sanketh

Reviewed By: sergei

Subscribers: mihnea, streddy, rthallam, bogdan, ybase

Differential Revision: https://phabricator.dev.yugabyte.com/D16415
es1024 added a commit that referenced this issue May 2, 2022
@es1024
[#11531] docdb: Added support for hot swapping postgres/CQLServer TLS…
1da19dc 
… certificates

Summary:
Modified the certificate reload path for tserver to also reload CQLServer and Postgres
(via SIGHUP) TLS certificates.

Test Plan:
`ybd --gtest_filter ExternalMiniClusterSecureTest.ReloadCertificates`
`ybd --java_test 'TestSecureCluster#testCertificateReload'`

Also verified that the certificates were updated connecting to :5433 and :9042 with openssl s_client.

Reviewers: sanketh, sergei

Reviewed By: sergei

Subscribers: mihnea, ybase, rthallam, bogdan

Differential Revision: https://phabricator.dev.yugabyte.com/D16596
@es1024
Copy link
Contributor

es1024 commented May 2, 2022

Added with 73a3ca4 and 1da19dc.

@es1024 es1024 closed this as completed May 2, 2022
@es1024 es1024 mentioned this issue May 2, 2022
Closed
@es1024 es1024 mentioned this issue May 10, 2022
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@bmatican bmatican

@es1024 es1024

Labels
area/docdb YugabyteDB core features
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@bmatican @spolitov @es1024 @rthallamko3 @streddy-yb