Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Admission control webhook with self-signed CA in "caBundle" is not trusted #61171

Closed
johanot opened this issue Mar 14, 2018 · 10 comments
Closed

Admission control webhook with self-signed CA in "caBundle" is not trusted #61171

johanot opened this issue Mar 14, 2018 · 10 comments

Comments

@johanot
Copy link

@johanot johanot commented Mar 14, 2018

/kind bug

What happened:

I have compiled and deployed the test admission webhook here: https://github.com/kubernetes/kubernetes/tree/master/test/images/webhook
.. into my cluster.

I've added a ValidatingWebhookConfiguration to my cluster intercepting namespace creation, just for initial testing:

apiVersion: admissionregistration.k8s.io/v1beta1
kind: ValidatingWebhookConfiguration
metadata:
  name: admission
webhooks:
- name: pod.admission.cloud
  rules:
  - apiGroups:
    - "*"
    apiVersions:
    - "*"
    operations:
    - "*"
    resources:
    - namespaces
  failurePolicy: Fail
  clientConfig:
    caBundle: MIIDSjCCAjKgAwIBAgIUWxmj40l+TDVJq98Xy7c6Leo3np8wDQYJKoZIhvcNAQELBQAwPTELMAkGA1UEBhMCeHgxCjAIBgNVBAgTAXgxCjAIBgNVBAcTAXgxCjAIBgNVBAoTAXgxCjAIBgNVBAsTAXgwHhcNMTgwMjAyMTIzODAwWhcNMjMwMjAxMTIzODAwWjA9MQswCQYDVQQGEwJ4eDEKMAgGA1UECBMBeDEKMAgGA1UEBxMBeDEKMAgGA1UEChMBeDEKMAgGA1UECxMBeDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBANHkqOmVf23KMXdaZU2eFUx1h4wb09JINBB8x/HL7UE0KFJcnOoVnNQB0gRukUopiYCzrzMFyGWWmB/pAEKool+ZiI2uMy6mcYBDtOi4pOm7U0TQQMV6L/5Yfi65xRz3RTMd/tYAoFi4aCZbJAGjxU6UWNYDzTy8E/cP6ZnlNbVHRiA6/wHsoWcXtWTXYP5yn9cf7EWQi1hOBM4BWmOIyB1f6LEgQipZWMOMPPHO3hsuSBn0rk7jovSt5XTlbgRrtxqAJiNjJUykWzIF+lLnZCioippGv5vkdGvE83JoACXvZTUwzA+MLu49fkw3bweqkbhrer8kacjfGlw3aJN37eECAwEAAaNCMEAwDgYDVR0PAQH/BAQDAgEGMA8GA1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYEFKXcb52bv6oqnD+D9fTNFHZL8IWxMA0GCSqGSIb3DQEBCwUAA4IBAQADvKvv3ym0XAYwKxPLLl3Lc6sJYHDbTN0donduG7PXeb1dhuukJ2lfufUYp2IGSAxuLecTYeeByOVp1gaMb5LsIGt2BVDmlMMkiH29LUHsvbyi85CpJo7A5RJG6AWW2VBCiDjz5v8JFM6pMkBRFfXH+pwIge65CE+MTSQcfb1/aIIoQ226P7E/3uUGX4k4pDXG/O7GNvykF40v1DB5y7DDBTQ4JWiJfyGkT69TmdOGLFAmjwxUjWyvEey4qJex/EGEm5RQcMv9iy7tba1wK7sykNGn5uDELGPGIIEAa5rIHm1FUFOZZVoELaasWS559wy8og39Eq21dDMynb8Bndn/
    service:
      name: admission
      namespace: default

When trying to create a namespace, admission-control fails with PKI-error (seen below), although the server cert is indeed signed with the certificate given in "caBundle":

Mar 14 11:45:15 control-plane-p01 kube-apiserver[25281]: W0314 11:45:15.177435 25281 admission.go:257] Failed calling webhook, failing closed pod.admission.cloud: failed calling admission webhook "pod.admission.cloud": Post https://admission.default.svc:443/: x509: certificate signed by unknown authority

Admission webhook log:

2018/03/14 11:48:42 http: TLS handshake error from 172.20.32.11:33616: remote error: tls: bad certificate

What you expected to happen:
apiserver should see my x509-chain as valid.

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Checking the cert chain with OpenSSL s_client works fine:

openssl s_client -connect admission.default.svc:443 -CAfile ca.pem
...
Verify return code: 0 (ok)

.. by the way.. Shouldn't I be able to add multiple certificates to the caBundle? If yes, how?

Environment:

  • Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.1", GitCommit:"3a1c9449a956b6026f075fa3134ff92f7d55f812", GitTreeState:"clean", BuildDate:"2018-01-04T11:52:23Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}

Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.4", GitCommit:"bee2d1505c4fe820744d26d41ecd3fdd4a3d6546", GitTreeState:"archive", BuildDate:"1970-01-01T00:00:01Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}

  • Cloud provider or hardware configuration:

Self hosted. Kubernetes build from source on Nix 17.09.

  • OS (e.g. from /etc/os-release):
  • Kernel (e.g. uname -a):

Linux container-p01 4.15.7 #1-NixOS SMP Wed Feb 28 09:21:39 UTC 2018 x86_64 GNU/Linux

  • Install tools:
  • Others:

go version go1.9.2 linux/amd64

@johanot

This comment has been minimized.

Copy link
Author

@johanot johanot commented Mar 14, 2018

/sig api-machinery

@liggitt

This comment has been minimized.

Copy link
Member

@liggitt liggitt commented Mar 14, 2018

Create a self-signed CA-certificate

Can you include the X509v3 extensions section of your CA?

openssl x509 -in /path/to/ca.crt -text -noout | grep "X509v3 extensions" -A 10

Shouldn't I be able to add multiple certificates to the caBundle? If yes, how?

Yes, the bundle can contain multiple PEM certificate blocks

@liggitt

This comment has been minimized.

Copy link
Member

@liggitt liggitt commented Mar 14, 2018

nevermind, I missed you included the ca bundle in the description. It needs to be in PEM format:

https://github.com/kubernetes/api/blob/master/admissionregistration/v1beta1/types.go#L264-L267

@liggitt

This comment has been minimized.

Copy link
Member

@liggitt liggitt commented Mar 14, 2018

/close

@johanot

This comment has been minimized.

Copy link
Author

@johanot johanot commented Mar 14, 2018

@liggitt Thanks, and perhaps pardon my stupidity. It is PEM, but without header and footer in my example, because... Adding:

-----BEGIN CERTIFICATE-----

.. gives decode base64: illegal base64 data at input byte 0

.. so it doesn't sound like the parser likes anything other than base64?

Can you please provide an example of how you'd encode it in yaml or json?

@liggitt

This comment has been minimized.

Copy link
Member

@liggitt liggitt commented Mar 14, 2018

to convert from DER to PEM, do openssl x509 -in ca.crt -noout -inform der -out ca.crt.pem

-----BEGIN CERTIFICATE-----
MIIDSjCCAjKgAwIBAgIUWxmj40l+TDVJq98Xy7c6Leo3np8wDQYJKoZIhvcNAQEL
BQAwPTELMAkGA1UEBhMCeHgxCjAIBgNVBAgTAXgxCjAIBgNVBAcTAXgxCjAIBgNV
BAoTAXgxCjAIBgNVBAsTAXgwHhcNMTgwMjAyMTIzODAwWhcNMjMwMjAxMTIzODAw
WjA9MQswCQYDVQQGEwJ4eDEKMAgGA1UECBMBeDEKMAgGA1UEBxMBeDEKMAgGA1UE
ChMBeDEKMAgGA1UECxMBeDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB
ANHkqOmVf23KMXdaZU2eFUx1h4wb09JINBB8x/HL7UE0KFJcnOoVnNQB0gRukUop
iYCzrzMFyGWWmB/pAEKool+ZiI2uMy6mcYBDtOi4pOm7U0TQQMV6L/5Yfi65xRz3
RTMd/tYAoFi4aCZbJAGjxU6UWNYDzTy8E/cP6ZnlNbVHRiA6/wHsoWcXtWTXYP5y
n9cf7EWQi1hOBM4BWmOIyB1f6LEgQipZWMOMPPHO3hsuSBn0rk7jovSt5XTlbgRr
txqAJiNjJUykWzIF+lLnZCioippGv5vkdGvE83JoACXvZTUwzA+MLu49fkw3bweq
kbhrer8kacjfGlw3aJN37eECAwEAAaNCMEAwDgYDVR0PAQH/BAQDAgEGMA8GA1Ud
EwEB/wQFMAMBAf8wHQYDVR0OBBYEFKXcb52bv6oqnD+D9fTNFHZL8IWxMA0GCSqG
SIb3DQEBCwUAA4IBAQADvKvv3ym0XAYwKxPLLl3Lc6sJYHDbTN0donduG7PXeb1d
huukJ2lfufUYp2IGSAxuLecTYeeByOVp1gaMb5LsIGt2BVDmlMMkiH29LUHsvbyi
85CpJo7A5RJG6AWW2VBCiDjz5v8JFM6pMkBRFfXH+pwIge65CE+MTSQcfb1/aIIo
Q226P7E/3uUGX4k4pDXG/O7GNvykF40v1DB5y7DDBTQ4JWiJfyGkT69TmdOGLFAm
jwxUjWyvEey4qJex/EGEm5RQcMv9iy7tba1wK7sykNGn5uDELGPGIIEAa5rIHm1F
UFOZZVoELaasWS559wy8og39Eq21dDMynb8Bndn/
-----END CERTIFICATE-----

that PEM blob goes in the caBundle, which is a []byte and so base64-encoded again in JSON :-/

caBundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURTakNDQWpLZ0F3SUJBZ0lVV3htajQwbCtURFZKcTk4WHk3YzZMZW8zbnA4d0RRWUpLb1pJaHZjTkFRRUwKQlFBd1BURUxNQWtHQTFVRUJoTUNlSGd4Q2pBSUJnTlZCQWdUQVhneENqQUlCZ05WQkFjVEFYZ3hDakFJQmdOVgpCQW9UQVhneENqQUlCZ05WQkFzVEFYZ3dIaGNOTVRnd01qQXlNVEl6T0RBd1doY05Nak13TWpBeE1USXpPREF3CldqQTlNUXN3Q1FZRFZRUUdFd0o0ZURFS01BZ0dBMVVFQ0JNQmVERUtNQWdHQTFVRUJ4TUJlREVLTUFnR0ExVUUKQ2hNQmVERUtNQWdHQTFVRUN4TUJlRENDQVNJd0RRWUpLb1pJaHZjTkFRRUJCUUFEZ2dFUEFEQ0NBUW9DZ2dFQgpBTkhrcU9tVmYyM0tNWGRhWlUyZUZVeDFoNHdiMDlKSU5CQjh4L0hMN1VFMEtGSmNuT29Wbk5RQjBnUnVrVW9wCmlZQ3pyek1GeUdXV21CL3BBRUtvb2wrWmlJMnVNeTZtY1lCRHRPaTRwT203VTBUUVFNVjZMLzVZZmk2NXhSejMKUlRNZC90WUFvRmk0YUNaYkpBR2p4VTZVV05ZRHpUeThFL2NQNlpubE5iVkhSaUE2L3dIc29XY1h0V1RYWVA1eQpuOWNmN0VXUWkxaE9CTTRCV21PSXlCMWY2TEVnUWlwWldNT01QUEhPM2hzdVNCbjByazdqb3ZTdDVYVGxiZ1JyCnR4cUFKaU5qSlV5a1d6SUYrbExuWkNpb2lwcEd2NXZrZEd2RTgzSm9BQ1h2WlRVd3pBK01MdTQ5Zmt3M2J3ZXEKa2JocmVyOGthY2pmR2x3M2FKTjM3ZUVDQXdFQUFhTkNNRUF3RGdZRFZSMFBBUUgvQkFRREFnRUdNQThHQTFVZApFd0VCL3dRRk1BTUJBZjh3SFFZRFZSME9CQllFRktYY2I1MmJ2Nm9xbkQrRDlmVE5GSFpMOElXeE1BMEdDU3FHClNJYjNEUUVCQ3dVQUE0SUJBUUFEdkt2djN5bTBYQVl3S3hQTExsM0xjNnNKWUhEYlROMGRvbmR1RzdQWGViMWQKaHV1a0oybGZ1ZlVZcDJJR1NBeHVMZWNUWWVlQnlPVnAxZ2FNYjVMc0lHdDJCVkRtbE1Na2lIMjlMVUhzdmJ5aQo4NUNwSm83QTVSSkc2QVdXMlZCQ2lEano1djhKRk02cE1rQlJGZlhIK3B3SWdlNjVDRStNVFNRY2ZiMS9hSUlvClEyMjZQN0UvM3VVR1g0azRwRFhHL083R052eWtGNDB2MURCNXk3RERCVFE0SldpSmZ5R2tUNjlUbWRPR0xGQW0Kand4VWpXeXZFZXk0cUpleC9FR0VtNVJRY012OWl5N3RiYTF3SzdzeWtOR241dURFTEdQR0lJRUFhNXJJSG0xRgpVRk9aWlZvRUxhYXNXUzU1OXd5OG9nMzlFcTIxZERNeW5iOEJuZG4vCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
@johanot

This comment has been minimized.

Copy link
Author

@johanot johanot commented Mar 14, 2018

Works perfectly! Missed the "double-encoding" part, and I think the docs confused me a bit (https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.9/#webhookclientconfig-v1beta1-admissionregistration), where it says type = string.

It is in fact:

  • a binary certificate
  • represented as a base64-encoded string
  • interpreted as a binary blob
  • transfer-encoded as a base64 string

:) thanks a lot again!

@daurnimator

This comment has been minimized.

Copy link

@daurnimator daurnimator commented Jul 30, 2018

FWIW I just wasted a few hours on this same issue: requiring double encoding is weird!

@lovejoy

This comment has been minimized.

Copy link
Contributor

@lovejoy lovejoy commented Dec 20, 2018

@TommyLike

This comment has been minimized.

Copy link

@TommyLike TommyLike commented Feb 1, 2019

mark

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
6 participants
You can’t perform that action at this time.