Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dynamic SNI certificates #84303

Merged
merged 1 commit into from Oct 31, 2019
Merged

Conversation

@jackkleeman
Copy link
Member

jackkleeman commented Oct 24, 2019

Based on #84200
/kind feature

What this PR does / why we need it:
This PR means that apiserver SNI certificates are reloaded off of disk every minute, and notify the dynamic certificate controller when they change, leading to an updated tls.Config being served.

Which issue(s) this PR fixes:
Fixes #66448

Does this PR introduce a user-facing change?:

Reload apiserver SNI certificates from disk every minute
@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

k8s-ci-robot commented Oct 24, 2019

Hi @jackkleeman. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@jackkleeman jackkleeman changed the title Dynamic sni certificates Dynamic SNI certificates Oct 24, 2019
@jackkleeman

This comment has been minimized.

Copy link
Member Author

jackkleeman commented Oct 24, 2019

/assign @deads2k

t.Fatal(err)
}

// when we run this the second time, we know which one we are expecting

This comment has been minimized.

Copy link
@johnSchnake

johnSchnake Oct 25, 2019

Contributor

nit: Prefer comments with proper capitalization/punctuation. Applies throughout.

t.Fatal(err)
}

dynamiccertificates.FileRefreshDuration = 1 * time.Second

This comment has been minimized.

Copy link
@johnSchnake

johnSchnake Oct 25, 2019

Contributor

Seems most tests that care about this value just set the time to 1s like this; clearly the global value could cause problems if tests needed different timings and tests may be in parallel. Not an issue in most cases and each test seems to be setting it as needed instead of assuming defaults. OK/consistent with current practice but wish it wasn't just a global value (not your problem to fix here)

tlsConfig := &tls.Config{
InsecureSkipVerify: true,
VerifyPeerCertificate: func(rawCerts [][]byte, verifiedChains [][]*x509.Certificate) error {
acceptableCerts = make([][]byte, 0, len(rawCerts))

This comment has been minimized.

Copy link
@johnSchnake

johnSchnake Oct 25, 2019

Contributor

We make 2 connections with this tlsConfig, so doesn't the 2nd time we run it it overwrites any info found via the first connection?

What is the point of conn then since we don't test anything about the acceptableCerts until after the conn2 is made?

Seems like after the first connection is made that you want to check that there are 0 certs?

This comment has been minimized.

Copy link
@jackkleeman

jackkleeman Oct 25, 2019

Author Member

I'll add a test on the initial connection's cert

if err != nil {
t.Fatal(err)
}
defer conn.Close()

This comment has been minimized.

Copy link
@johnSchnake

johnSchnake Oct 25, 2019

Contributor

Is there a reason not to just close it right away? We don't actually need the connection for anything, just using the tlsConfig to grab cert data, right? Same for conn2

@johnSchnake

This comment has been minimized.

Copy link
Contributor

johnSchnake commented Oct 25, 2019

/ok-to-test

Reload SNI certificate cert and key file from disk every minute and notify
the dynamic certificate controller when they change, allowing serving
tls config to be updated.
@jackkleeman jackkleeman force-pushed the jackkleeman:dynamic-sni-cert branch from 033170c to d9adf53 Oct 25, 2019
@jackkleeman

This comment has been minimized.

Copy link
Member Author

jackkleeman commented Oct 25, 2019

Thanks @johnSchnake, updated

@deads2k

This comment has been minimized.

Copy link
Contributor

deads2k commented Oct 30, 2019

this fell out extremely well.

/lgtm
/approve

@k8s-ci-robot k8s-ci-robot added the lgtm label Oct 30, 2019
@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

k8s-ci-robot commented Oct 30, 2019

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: deads2k, jackkleeman

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@fejta-bot

This comment has been minimized.

Copy link

fejta-bot commented Oct 31, 2019

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

1 similar comment
@fejta-bot

This comment has been minimized.

Copy link

fejta-bot commented Oct 31, 2019

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

@k8s-ci-robot k8s-ci-robot merged commit c7bb076 into kubernetes:master Oct 31, 2019
16 checks passed
16 checks passed
cla/linuxfoundation jackkleeman authorized
Details
pull-kubernetes-bazel-build Job succeeded.
Details
pull-kubernetes-bazel-test Job succeeded.
Details
pull-kubernetes-conformance-kind-ipv6 Job succeeded.
Details
pull-kubernetes-dependencies Job succeeded.
Details
pull-kubernetes-e2e-gce Job succeeded.
Details
pull-kubernetes-e2e-gce-100-performance Job succeeded.
Details
pull-kubernetes-e2e-gce-device-plugin-gpu Job succeeded.
Details
pull-kubernetes-e2e-kind Job succeeded.
Details
pull-kubernetes-integration Job succeeded.
Details
pull-kubernetes-kubemark-e2e-gce-big Job succeeded.
Details
pull-kubernetes-node-e2e Job succeeded.
Details
pull-kubernetes-node-e2e-containerd Job succeeded.
Details
pull-kubernetes-typecheck Job succeeded.
Details
pull-kubernetes-verify Job succeeded.
Details
tide In merge pool.
Details
@k8s-ci-robot k8s-ci-robot added this to the v1.17 milestone Oct 31, 2019
@jackkleeman jackkleeman deleted the jackkleeman:dynamic-sni-cert branch Oct 31, 2019
@@ -251,7 +251,7 @@ func (s *SecureServingOptions) ApplyTo(config **server.SecureServingInfo) error
// load SNI certs
namedTLSCerts := make([]dynamiccertificates.SNICertKeyContentProvider, 0, len(s.SNICertKeys))
for _, nck := range s.SNICertKeys {
tlsCert, err := dynamiccertificates.NewStaticSNICertKeyContentFromFiles(nck.CertFile, nck.KeyFile, nck.Names...)
tlsCert, err := dynamiccertificates.NewDynamicSNIContentFromFiles("sni-serving-cert", nck.CertFile, nck.KeyFile, nck.Names...)

This comment has been minimized.

Copy link
@dminca

dminca Nov 6, 2019

why does "sni-serving-cert" have to be hardcoded? 💭

@jackkleeman jackkleeman mentioned this pull request Nov 12, 2019
6 of 6 tasks complete
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.