Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix client-ca dynamic reload in apiserver #90360

Merged
merged 1 commit into from Apr 30, 2020
Merged

Fix client-ca dynamic reload in apiserver #90360

merged 1 commit into from Apr 30, 2020

Conversation

tnozicka
Copy link
Contributor

@tnozicka tnozicka commented Apr 22, 2020 •
edited

What type of PR is this?
/kind bug

What this PR does / why we need it:
kube-apiserver fails to live reload changed client CA and front proxy CA and refuses clients signed by the new CA with 401 Unable to authenticate the request due to an error: x509: certificate signed by unknown authority

Which issue(s) this PR fixes:

Special notes for your reviewer:
The dynamic reload code is already there, some instances of those controllers were never started though. (See #90360 (comment) and #90360 (comment)). Starting the auto-reload only for some instances of those controllers explains why the current test only checking published CAs was succeeding. This PR is extending the test to actually test the authentication flow for both CAs.

Does this PR introduce a user-facing change?:

NONE

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

/cc @deads2k
@kubernetes/sig-api-machinery-pr-reviews

@k8s-ci-robot k8s-ci-robot added release-note-none Denotes a PR that doesn't merit a release note. sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. kind/bug Categorizes issue or PR as related to a bug. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels Apr 22, 2020
@tnozicka
Copy link
Contributor Author

/priority important-soon

@k8s-ci-robot k8s-ci-robot added priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. area/apiserver area/kubelet area/test sig/node Categorizes an issue or PR as relevant to SIG Node. sig/testing Categorizes an issue or PR as relevant to SIG Testing. and removed needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels Apr 22, 2020
deads2k
deads2k reviewed Apr 22, 2020
View reviewed changes
staging/src/k8s.io/apiserver/pkg/server/options/authentication.go Outdated
if err != nil {
return nil, err
}
go clientCAProvider.Run(1, context.TODO().Done())
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the test demonstrated that the server properly updated the requested certificates. Was the test flawed? If so, it would be great to see a fix to the flawed test first without your changes so we can see it fail.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The test wasn't testing the certificates are actually in use, just that they report to be. The new test adds a client call actually testing the authentication.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

3 instances of the client-ca were created only 2 were started

I0423 08:29:39.620652  967521 dynamic_cafile_content.go:102] === NewDynamicCAContentFromFile client-ca-bundle::/tmp/test-integration-TestClientCA158575615/client-ca.crt180811401(0xc0000b0e40)
goroutine 57 [running]:
runtime/debug.Stack(0xc0000b0e40, 0x0, 0x0)
	/home/tnozicka/lib/go1.14.2/src/runtime/debug/stack.go:24 +0xab
k8s.io/apiserver/pkg/server/dynamiccertificates.NewDynamicCAContentFromFile(0x5eccff8, 0x10, 0xc000a207d0, 0x42, 0x1, 0x1, 0x71f7240)
	/home/tnozicka/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/dynamiccertificates/dynamic_cafile_content.go:102 +0x6d2
k8s.io/apiserver/pkg/server/options.(*ClientCertAuthenticationOptions).GetClientCAContentProvider(0xc00000f100, 0x11, 0x5eb030a, 0x0, 0x71f7240)
	/home/tnozicka/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/options/authentication.go:151 +0xa9
k8s.io/kubernetes/pkg/kubeapiserver/options.(*BuiltInAuthenticationOptions).ApplyTo(0xc000ae2150, 0xc000908480, 0xc0009084c0, 0x0)
	/home/tnozicka/go/src/k8s.io/kubernetes/pkg/kubeapiserver/options/authentication.go:408 +0x6e0
k8s.io/kubernetes/cmd/kube-apiserver/app.buildGenericConfig(0xc00055c000, 0xc0008a43c0, 0x0, 0x0, 0xc000a23520, 0x0, 0x0, 0xd0, 0xd0, 0xc000a23520, ...)
	/home/tnozicka/go/src/k8s.io/kubernetes/cmd/kube-apiserver/app/server.go:446 +0x721
k8s.io/kubernetes/cmd/kube-apiserver/app.CreateKubeAPIServerConfig(0xc00055c000, 0x0, 0x0, 0xc0008a43c0, 0x0, 0x0, 0x71f8780, 0xc00000ed20, 0x989b, 0x0, ...)
	/home/tnozicka/go/src/k8s.io/kubernetes/cmd/kube-apiserver/app/server.go:287 +0x61
k8s.io/kubernetes/test/integration/framework.StartTestServer(0xc0002899e0, 0xc000116660, 0xc0007073f0, 0x0, 0x0, 0x0, 0x0)
	/home/tnozicka/go/src/k8s.io/kubernetes/test/integration/framework/test_server.go:117 +0x152d
k8s.io/kubernetes/test/integration/apiserver/certreload.TestClientCA(0xc0002899e0)
	/home/tnozicka/go/src/k8s.io/kubernetes/test/integration/apiserver/certreload/certreload_test.go:169 +0x630
testing.tRunner(0xc0002899e0, 0x6052bf0)
	/home/tnozicka/lib/go1.14.2/src/testing/testing.go:991 +0x1ec
created by testing.(*T).Run
	/home/tnozicka/lib/go1.14.2/src/testing/testing.go:1042 +0x661


I0423 08:29:39.630980  967521 dynamic_cafile_content.go:102] === NewDynamicCAContentFromFile client-ca-bundle::/tmp/test-integration-TestClientCA158575615/client-ca.crt180811401(0xc0004bd380)
goroutine 57 [running]:
runtime/debug.Stack(0xc0004bd380, 0x0, 0x0)
	/home/tnozicka/lib/go1.14.2/src/runtime/debug/stack.go:24 +0xab
k8s.io/apiserver/pkg/server/dynamiccertificates.NewDynamicCAContentFromFile(0x5eccff8, 0x10, 0xc000a207d0, 0x42, 0x0, 0x0, 0x5f243e5)
	/home/tnozicka/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/dynamiccertificates/dynamic_cafile_content.go:102 +0x6d2
k8s.io/apiserver/pkg/server/options.(*ClientCertAuthenticationOptions).GetClientCAContentProvider(0xc00000f100, 0x24, 0x0, 0x0, 0x0)
	/home/tnozicka/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/options/authentication.go:151 +0xa9
k8s.io/kubernetes/pkg/kubeapiserver/options.(*BuiltInAuthenticationOptions).ToAuthenticationConfig(0xc000ae2150, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/home/tnozicka/go/src/k8s.io/kubernetes/pkg/kubeapiserver/options/authentication.go:344 +0xe10
k8s.io/kubernetes/cmd/kube-apiserver/app.BuildAuthenticator(0xc00055c000, 0x0, 0x72a18e0, 0xc0002caf20, 0x7289cc0, 0xc00004fea0, 0x7289cc0, 0xc00004fea0, 0x0, 0x0, ...)
	/home/tnozicka/go/src/k8s.io/kubernetes/cmd/kube-apiserver/app/server.go:567 +0x77
k8s.io/kubernetes/cmd/kube-apiserver/app.buildGenericConfig(0xc00055c000, 0xc0008a43c0, 0x0, 0x0, 0xc000a23520, 0x0, 0x0, 0xd0, 0xd0, 0xc000a23520, ...)
	/home/tnozicka/go/src/k8s.io/kubernetes/cmd/kube-apiserver/app/server.go:504 +0x12fa
k8s.io/kubernetes/cmd/kube-apiserver/app.CreateKubeAPIServerConfig(0xc00055c000, 0x0, 0x0, 0xc0008a43c0, 0x0, 0x0, 0x71f8780, 0xc00000ed20, 0x989b, 0x0, ...)
	/home/tnozicka/go/src/k8s.io/kubernetes/cmd/kube-apiserver/app/server.go:287 +0x61
k8s.io/kubernetes/test/integration/framework.StartTestServer(0xc0002899e0, 0xc000116660, 0xc0007073f0, 0x0, 0x0, 0x0, 0x0)
	/home/tnozicka/go/src/k8s.io/kubernetes/test/integration/framework/test_server.go:117 +0x152d
k8s.io/kubernetes/test/integration/apiserver/certreload.TestClientCA(0xc0002899e0)
	/home/tnozicka/go/src/k8s.io/kubernetes/test/integration/apiserver/certreload/certreload_test.go:169 +0x630
testing.tRunner(0xc0002899e0, 0x6052bf0)
	/home/tnozicka/lib/go1.14.2/src/testing/testing.go:991 +0x1ec
created by testing.(*T).Run
	/home/tnozicka/lib/go1.14.2/src/testing/testing.go:1042 +0x661
I


I0423 08:29:39.637074  967521 dynamic_cafile_content.go:102] === NewDynamicCAContentFromFile client-ca-bundle::/tmp/test-integration-TestClientCA158575615/client-ca.crt180811401(0xc000112fc0)
goroutine 57 [running]:
runtime/debug.Stack(0xc000112fc0, 0x0, 0x0)
	/home/tnozicka/lib/go1.14.2/src/runtime/debug/stack.go:24 +0xab
k8s.io/apiserver/pkg/server/dynamiccertificates.NewDynamicCAContentFromFile(0x5eccff8, 0x10, 0xc000a207d0, 0x42, 0xc000a2fd66, 0xc000706630, 0xc000caec00)
	/home/tnozicka/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/dynamiccertificates/dynamic_cafile_content.go:102 +0x6d2
k8s.io/apiserver/pkg/server/options.(*ClientCertAuthenticationOptions).GetClientCAContentProvider(0xc00000f100, 0x10, 0x4, 0xc0000e0280, 0x4)
	/home/tnozicka/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/options/authentication.go:151 +0xa9
k8s.io/kubernetes/cmd/kube-apiserver/app.CreateKubeAPIServerConfig(0xc00055c000, 0x0, 0x0, 0xc0008a43c0, 0x0, 0x0, 0x71f8780, 0xc00000ed20, 0x989b, 0x0, ...)
	/home/tnozicka/go/src/k8s.io/kubernetes/cmd/kube-apiserver/app/server.go:359 +0xb48
k8s.io/kubernetes/test/integration/framework.StartTestServer(0xc0002899e0, 0xc000116660, 0xc0007073f0, 0x0, 0x0, 0x0, 0x0)
	/home/tnozicka/go/src/k8s.io/kubernetes/test/integration/framework/test_server.go:117 +0x152d
k8s.io/kubernetes/test/integration/apiserver/certreload.TestClientCA(0xc0002899e0)
	/home/tnozicka/go/src/k8s.io/kubernetes/test/integration/apiserver/certreload/certreload_test.go:169 +0x630
testing.tRunner(0xc0002899e0, 0x6052bf0)
	/home/tnozicka/lib/go1.14.2/src/testing/testing.go:991 +0x1ec
created by testing.(*T).Run
	/home/tnozicka/lib/go1.14.2/src/testing/testing.go:1042 +0x661


I0423 08:29:45.215943  967521 dynamic_cafile_content.go:167] === DynamicFileCAContent.Run client-ca-bundle::/tmp/test-integration-TestClientCA158575615/client-ca.crt180811401(0xc0000b0e40)
goroutine 1516 [running]:
runtime/debug.Stack(0xc0000b0e40, 0x0, 0xc000000001)
	/home/tnozicka/lib/go1.14.2/src/runtime/debug/stack.go:24 +0xab
k8s.io/apiserver/pkg/server/dynamiccertificates.(*DynamicFileCAContent).Run(0xc0000b0e40, 0x1, 0xc000becf60)
	/home/tnozicka/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/dynamiccertificates/dynamic_cafile_content.go:167 +0x77
created by k8s.io/apiserver/pkg/server/dynamiccertificates.unionCAContent.Run
	/home/tnozicka/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/dynamiccertificates/union_content.go:104 +0xec

I0423 08:29:45.225615  967521 dynamic_cafile_content.go:167] === DynamicFileCAContent.Run client-ca-bundle::/tmp/test-integration-TestClientCA158575615/client-ca.crt180811401(0xc000112fc0)
goroutine 1558 [running]:
runtime/debug.Stack(0xc000112fc0, 0x461e96, 0x203000)
	/home/tnozicka/lib/go1.14.2/src/runtime/debug/stack.go:24 +0xab
k8s.io/apiserver/pkg/server/dynamiccertificates.(*DynamicFileCAContent).Run(0xc000112fc0, 0x1, 0xc000becea0)
	/home/tnozicka/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/dynamiccertificates/dynamic_cafile_content.go:167 +0x77
created by k8s.io/kubernetes/pkg/master.completedConfig.New.func1
	/home/tnozicka/go/src/k8s.io/kubernetes/pkg/master/master.go:464 +0x42b

Copy link
Contributor Author

@tnozicka tnozicka Apr 23, 2020 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suppose the one that isn't started (from BuildAuthenticator) would explain why you get

E0423 08:29:51.872852  967521 authentication.go:53] Unable to authenticate the request due to an error: x509: certificate signed by unknown authority
    TestClientCA: certreload_test.go:292: Unauthorized

if you run the new test against the code from master. It also shows that the tests wasn't actually executing the authentication flow.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for the record same 2 of 3 for request header auth

I0423 10:55:51.364805  990025 dynamic_cafile_content.go:102] === NewDynamicCAContentFromFile request-header::/tmp/test-integration-TestClientCA126115273/proxy-ca.crt051635860(0xc000388c60)
goroutine 135 [running]:
runtime/debug.Stack(0xc000388c60, 0x0, 0x0)
	/home/tnozicka/lib/go1.14.2/src/runtime/debug/stack.go:24 +0xab
k8s.io/apiserver/pkg/server/dynamiccertificates.NewDynamicCAContentFromFile(0x5ec7123, 0xe, 0xc0004005f0, 0x41, 0x0, 0x0, 0xc000388840)
	/home/tnozicka/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/dynamiccertificates/dynamic_cafile_content.go:102 +0x6d2
k8s.io/apiserver/pkg/server/options.(*RequestHeaderAuthenticationOptions).ToAuthenticationRequestHeaderConfig(0xc000572150, 0x71f7500, 0xc000388a20, 0x0)
	/home/tnozicka/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/options/authentication.go:114 +0x87
k8s.io/kubernetes/pkg/kubeapiserver/options.(*BuiltInAuthenticationOptions).ApplyTo(0xc0005720e0, 0xc000944900, 0xc000944940, 0x0)
	/home/tnozicka/go/src/k8s.io/kubernetes/pkg/kubeapiserver/options/authentication.go:417 +0x343
k8s.io/kubernetes/cmd/kube-apiserver/app.buildGenericConfig(0xc00071a580, 0xc0008e2500, 0x0, 0x0, 0xc0009188f0, 0x0, 0x0, 0xd0, 0xd0, 0xc0009188f0, ...)
	/home/tnozicka/go/src/k8s.io/kubernetes/cmd/kube-apiserver/app/server.go:447 +0x721
k8s.io/kubernetes/cmd/kube-apiserver/app.CreateKubeAPIServerConfig(0xc00071a580, 0x0, 0x0, 0xc0008e2500, 0x0, 0x0, 0x71f8b00, 0xc0000d6a20, 0xa45f, 0x0, ...)
	/home/tnozicka/go/src/k8s.io/kubernetes/cmd/kube-apiserver/app/server.go:288 +0x61
k8s.io/kubernetes/test/integration/framework.StartTestServer(0xc0002b8a20, 0xc0009bc2a0, 0xc000a1f3f0, 0x0, 0x0, 0x0, 0x0)
	/home/tnozicka/go/src/k8s.io/kubernetes/test/integration/framework/test_server.go:117 +0x152d
k8s.io/kubernetes/test/integration/apiserver/certreload.TestClientCA(0xc0002b8a20)
	/home/tnozicka/go/src/k8s.io/kubernetes/test/integration/apiserver/certreload/certreload_test.go:169 +0x630
testing.tRunner(0xc0002b8a20, 0x6052dd8)
	/home/tnozicka/lib/go1.14.2/src/testing/testing.go:991 +0x1ec
created by testing.(*T).Run
	/home/tnozicka/lib/go1.14.2/src/testing/testing.go:1042 +0x661


I0423 10:55:51.375374  990025 dynamic_cafile_content.go:102] === NewDynamicCAContentFromFile request-header::/tmp/test-integration-TestClientCA126115273/proxy-ca.crt051635860(0xc00042bec0)
goroutine 135 [running]:
runtime/debug.Stack(0xc00042bec0, 0x0, 0x0)
	/home/tnozicka/lib/go1.14.2/src/runtime/debug/stack.go:24 +0xab
k8s.io/apiserver/pkg/server/dynamiccertificates.NewDynamicCAContentFromFile(0x5ec7123, 0xe, 0xc0004005f0, 0x41, 0x0, 0x0, 0xc000a1da40)
	/home/tnozicka/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/dynamiccertificates/dynamic_cafile_content.go:102 +0x6d2
k8s.io/apiserver/pkg/server/options.(*RequestHeaderAuthenticationOptions).ToAuthenticationRequestHeaderConfig(0xc000572150, 0x71f7500, 0xc00042bc80, 0x0)
	/home/tnozicka/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/options/authentication.go:114 +0x87
k8s.io/kubernetes/pkg/kubeapiserver/options.(*BuiltInAuthenticationOptions).ToAuthenticationConfig(0xc0005720e0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/home/tnozicka/go/src/k8s.io/kubernetes/pkg/kubeapiserver/options/authentication.go:364 +0xa88
k8s.io/kubernetes/cmd/kube-apiserver/app.BuildAuthenticator(0xc00071a580, 0x0, 0x72a1c60, 0xc00042c840, 0x728a040, 0xc000432aa0, 0x728a040, 0xc000432aa0, 0x0, 0x0, ...)
	/home/tnozicka/go/src/k8s.io/kubernetes/cmd/kube-apiserver/app/server.go:575 +0xc2
k8s.io/kubernetes/cmd/kube-apiserver/app.buildGenericConfig(0xc00071a580, 0xc0008e2500, 0x0, 0x0, 0xc0009188f0, 0x0, 0x0, 0xd0, 0xd0, 0xc0009188f0, ...)
	/home/tnozicka/go/src/k8s.io/kubernetes/cmd/kube-apiserver/app/server.go:506 +0x12fa
k8s.io/kubernetes/cmd/kube-apiserver/app.CreateKubeAPIServerConfig(0xc00071a580, 0x0, 0x0, 0xc0008e2500, 0x0, 0x0, 0x71f8b00, 0xc0000d6a20, 0xa45f, 0x0, ...)
	/home/tnozicka/go/src/k8s.io/kubernetes/cmd/kube-apiserver/app/server.go:288 +0x61
k8s.io/kubernetes/test/integration/framework.StartTestServer(0xc0002b8a20, 0xc0009bc2a0, 0xc000a1f3f0, 0x0, 0x0, 0x0, 0x0)
	/home/tnozicka/go/src/k8s.io/kubernetes/test/integration/framework/test_server.go:117 +0x152d
k8s.io/kubernetes/test/integration/apiserver/certreload.TestClientCA(0xc0002b8a20)
	/home/tnozicka/go/src/k8s.io/kubernetes/test/integration/apiserver/certreload/certreload_test.go:169 +0x630
testing.tRunner(0xc0002b8a20, 0x6052dd8)
	/home/tnozicka/lib/go1.14.2/src/testing/testing.go:991 +0x1ec
created by testing.(*T).Run
	/home/tnozicka/lib/go1.14.2/src/testing/testing.go:1042 +0x661


I0423 10:55:51.382574  990025 dynamic_cafile_content.go:102] === NewDynamicCAContentFromFile request-header::/tmp/test-integration-TestClientCA126115273/proxy-ca.crt051635860(0xc0000aba40)
goroutine 135 [running]:
runtime/debug.Stack(0xc0000aba40, 0x0, 0x0)
	/home/tnozicka/lib/go1.14.2/src/runtime/debug/stack.go:24 +0xab
k8s.io/apiserver/pkg/server/dynamiccertificates.NewDynamicCAContentFromFile(0x5ec7123, 0xe, 0xc0004005f0, 0x41, 0x0, 0x0, 0xc000a1e630)
	/home/tnozicka/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/dynamiccertificates/dynamic_cafile_content.go:102 +0x6d2
k8s.io/apiserver/pkg/server/options.(*RequestHeaderAuthenticationOptions).ToAuthenticationRequestHeaderConfig(0xc000572150, 0x71f7500, 0xc0000ab740, 0x0)
	/home/tnozicka/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/options/authentication.go:114 +0x87
k8s.io/kubernetes/cmd/kube-apiserver/app.CreateKubeAPIServerConfig(0xc00071a580, 0x0, 0x0, 0xc0008e2500, 0x0, 0x0, 0x71f8b00, 0xc0000d6a20, 0xa45f, 0x0, ...)
	/home/tnozicka/go/src/k8s.io/kubernetes/cmd/kube-apiserver/app/server.go:366 +0xc00
k8s.io/kubernetes/test/integration/framework.StartTestServer(0xc0002b8a20, 0xc0009bc2a0, 0xc000a1f3f0, 0x0, 0x0, 0x0, 0x0)
	/home/tnozicka/go/src/k8s.io/kubernetes/test/integration/framework/test_server.go:117 +0x152d
k8s.io/kubernetes/test/integration/apiserver/certreload.TestClientCA(0xc0002b8a20)
	/home/tnozicka/go/src/k8s.io/kubernetes/test/integration/apiserver/certreload/certreload_test.go:169 +0x630
testing.tRunner(0xc0002b8a20, 0x6052dd8)
	/home/tnozicka/lib/go1.14.2/src/testing/testing.go:991 +0x1ec
created by testing.(*T).Run
	/home/tnozicka/lib/go1.14.2/src/testing/testing.go:1042 +0x661




I0423 10:55:56.955038  990025 dynamic_cafile_content.go:167] === DynamicFileCAContent.Run request-header::/tmp/test-integration-TestClientCA126115273/proxy-ca.crt051635860(0xc000388c60)
goroutine 1498 [running]:
runtime/debug.Stack(0xc000388c60, 0x0, 0x0)
	/home/tnozicka/lib/go1.14.2/src/runtime/debug/stack.go:24 +0xab
k8s.io/apiserver/pkg/server/dynamiccertificates.(*DynamicFileCAContent).Run(0xc000388c60, 0x1, 0xc00b1d4600)
	/home/tnozicka/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/dynamiccertificates/dynamic_cafile_content.go:167 +0x77
created by k8s.io/apiserver/pkg/server/dynamiccertificates.unionCAContent.Run
	/home/tnozicka/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/dynamiccertificates/union_content.go:104 +0xec

I0423 10:55:56.976483  990025 dynamic_cafile_content.go:167] === DynamicFileCAContent.Run request-header::/tmp/test-integration-TestClientCA126115273/proxy-ca.crt051635860(0xc0000aba40)
goroutine 1534 [running]:
runtime/debug.Stack(0xc0000aba40, 0x0, 0x0)
	/home/tnozicka/lib/go1.14.2/src/runtime/debug/stack.go:24 +0xab
k8s.io/apiserver/pkg/server/dynamiccertificates.(*DynamicFileCAContent).Run(0xc0000aba40, 0x1, 0xc00b1d4300)
	/home/tnozicka/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/dynamiccertificates/dynamic_cafile_content.go:167 +0x77
created by k8s.io/kubernetes/pkg/master.completedConfig.New.func1
	/home/tnozicka/go/src/k8s.io/kubernetes/pkg/master/master.go:476 +0x2dc

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

3 instances of the client-ca were created only 2 were started

This is an interesting result, seems like it may be the actual bug.

@tnozicka tnozicka changed the title Fix client-ca dynamic reload in apiserver [WIP] Fix client-ca dynamic reload in apiserver Apr 23, 2020
@k8s-ci-robot k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Apr 23, 2020
@tnozicka tnozicka force-pushed the fix-client-ca-reload branch 4 times, most recently from 5f305d6 to 563d569 Compare April 23, 2020 09:52
@tnozicka tnozicka changed the title [WIP] Fix client-ca dynamic reload in apiserver Fix client-ca dynamic reload in apiserver Apr 23, 2020
@k8s-ci-robot k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Apr 23, 2020
@tnozicka
Copy link
Contributor Author

tnozicka commented Apr 23, 2020 •
edited

verify got stuck
/test pull-kubernetes-verify

@k8s-ci-robot
Copy link
Contributor

@tnozicka: The specified target(s) for /test were not found.
The following commands are available to trigger jobs:

  • /test pull-kubernetes-bazel-build
  • /test pull-kubernetes-bazel-test
  • /test pull-kubernetes-conformance-image-test
  • /test pull-kubernetes-conformance-kind-ipv6-parallel
  • /test pull-kubernetes-dependencies
  • /test pull-kubernetes-dependencies-canary
  • /test pull-kubernetes-e2e-aws-eks-1-13-correctness
  • /test pull-kubernetes-e2e-gce
  • /test pull-kubernetes-e2e-gce-canary
  • /test pull-kubernetes-e2e-gce-ubuntu
  • /test pull-kubernetes-e2e-gce-ubuntu-containerd
  • /test pull-kubernetes-e2e-gce-rbe
  • /test pull-kubernetes-e2e-gce-alpha-features
  • /test pull-kubernetes-e2e-gce-device-plugin-gpu
  • /test pull-kubernetes-integration
  • /test pull-kubernetes-cross
  • /test pull-kubernetes-e2e-kind
  • /test pull-kubernetes-e2e-kind-canary
  • /test pull-kubernetes-e2e-kind-ipv6
  • /test pull-kubernetes-e2e-kind-ipv6-canary
  • /test pull-kubernetes-conformance-kind-ga-only
  • /test pull-kubernetes-conformance-kind-ga-only-parallel
  • /test pull-kubernetes-e2e-kops-aws
  • /test pull-kubernetes-bazel-build-canary
  • /test pull-kubernetes-bazel-test-canary
  • /test pull-kubernetes-bazel-test-integration-canary
  • /test pull-kubernetes-local-e2e
  • /test pull-publishing-bot-validate
  • /test pull-kubernetes-e2e-gce-network-proxy-http-connect
  • /test pull-kubernetes-e2e-gce-network-proxy-grpc
  • /test pull-kubernetes-e2e-gci-gce-autoscaling
  • /test pull-kubernetes-e2e-aks-engine-azure
  • /test pull-kubernetes-e2e-azure-disk
  • /test pull-kubernetes-e2e-azure-disk-vmss
  • /test pull-kubernetes-e2e-azure-file
  • /test pull-kubernetes-e2e-gci-gce-ipvs
  • /test pull-kubernetes-node-e2e
  • /test pull-kubernetes-e2e-containerd-gce
  • /test pull-kubernetes-node-e2e-containerd
  • /test pull-kubernetes-node-e2e-alpha
  • /test pull-kubernetes-node-kubelet-serial-cpu-manager
  • /test pull-kubernetes-node-kubelet-serial-topology-manager
  • /test pull-kubernetes-e2e-gce-100-performance
  • /test pull-kubernetes-e2e-gce-big-performance
  • /test pull-kubernetes-e2e-gce-large-performance
  • /test pull-kubernetes-kubemark-e2e-gce-big
  • /test pull-kubernetes-kubemark-e2e-gce-scale
  • /test pull-kubernetes-e2e-gce-storage-slow
  • /test pull-kubernetes-e2e-gce-storage-snapshot
  • /test pull-kubernetes-e2e-gce-storage-slow-rbe
  • /test pull-kubernetes-e2e-gce-csi-serial
  • /test pull-kubernetes-e2e-gce-iscsi
  • /test pull-kubernetes-e2e-gce-iscsi-serial
  • /test pull-kubernetes-e2e-gce-storage-disruptive
  • /test pull-kubernetes-e2e-aks-engine-azure-windows
  • /test pull-kubernetes-e2e-azure-disk-windows
  • /test pull-kubernetes-e2e-azure-file-windows
  • /test pull-kubernetes-typecheck
  • /test pull-kubernetes-verify
  • /test pull-kubernetes-e2e-windows-gce

Use /test all to run all jobs.

In response to this:

verify got stuck
/test verify

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@tnozicka
Copy link
Contributor Author

/test pull-kubernetes-verify

@deads2k
Copy link
Contributor

deads2k commented Apr 27, 2020

/approve

this lgtm, but I'd appreciate another set of eyes.

/assign @sttts

@tnozicka tnozicka force-pushed the fix-client-ca-reload branch 3 times, most recently from 02076a4 to 17f454d Compare April 29, 2020 13:07
@sttts
Copy link
Contributor

sttts commented Apr 29, 2020

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Apr 29, 2020
@derekwaynecarr
Copy link
Member

kubelet changes look good.

/approve
/lgtm

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: deads2k, derekwaynecarr, tnozicka

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Apr 29, 2020
@fejta-bot
Copy link

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

2 similar comments
@fejta-bot
Copy link

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

@fejta-bot
Copy link

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

@k8s-ci-robot
Copy link
Contributor

k8s-ci-robot commented Apr 30, 2020 •
edited

@tnozicka: The following test failed, say /retest to rerun all failed tests:

Test name Commit Details Rerun command
pull-kubernetes-e2e-kind-ipv6 b22a170 link /test pull-kubernetes-e2e-kind-ipv6

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@k8s-ci-robot k8s-ci-robot merged commit 8876563 into kubernetes:master Apr 30, 2020
@k8s-ci-robot k8s-ci-robot added this to the v1.19 milestone Apr 30, 2020
@tnozicka tnozicka deleted the fix-client-ca-reload branch May 4, 2020 07:21
This was referenced May 4, 2020
Merged
Merged
Merged
Merged
@johananl johananl mentioned this pull request Oct 20, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sttts sttts sttts left review comments

@deads2k deads2k deads2k left review comments

@p0lyn0mial p0lyn0mial p0lyn0mial left review comments

Assignees

@sttts sttts

@derekwaynecarr derekwaynecarr

@deads2k deads2k

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/apiserver area/kubelet area/test cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/bug Categorizes issue or PR as related to a bug. lgtm "Looks good to me", indicates that a PR is ready to be merged. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. release-note-none Denotes a PR that doesn't merit a release note. sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. sig/node Categorizes an issue or PR as relevant to SIG Node. sig/testing Categorizes an issue or PR as relevant to SIG Testing. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
None yet
Milestone
v1.19
Development

Successfully merging this pull request may close these issues.

None yet
8 participants
@tnozicka @k8s-ci-robot @deads2k @fedebongio @sttts @derekwaynecarr @fejta-bot @p0lyn0mial