Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Kubernetes Tutorial doesn't work #4902

Closed
Impakt opened this issue Jul 18, 2022 · 1 comment · Fixed by #4907
Closed

Kubernetes Tutorial doesn't work #4902

Impakt opened this issue Jul 18, 2022 · 1 comment · Fixed by #4907

Comments

@Impakt
Copy link

Impakt commented Jul 18, 2022

Short description

Hello,
I tried to follow the kubernetes tutorial here. If i run kubectl logs -l app=opa -c opa -f before I run kubectl apply -f webhook-configuration.yaml the logs appear fine. If I then run kubectl apply -f webhook-configuration.yaml in a separate terminal session, the logs start showing errors like this:

2022/07/18 15:59:20 http: TLS handshake error from 172.17.0.1:8112: remote error: tls: bad certificate
2022/07/18 15:59:20 http: TLS handshake error from 172.17.0.1:59081: remote error: tls: bad certificate
2022/07/18 15:59:20 http: TLS handshake error from 172.17.0.1:21681: remote error: tls: bad certificate
2022/07/18 15:59:20 http: TLS handshake error from 172.17.0.1:35477: remote error: tls: bad certificate

However, if i run kubectl apply -f webhook-configuration.yaml before I check the logs (as per the tutorial), i get Authorization errors a few seconds after running kubectl logs -l app=opa -c opa -f:

Error from server (InternalError): Internal error occurred: Authorization error (user=kube-apiserver-kubelet-client, verb=get, resource=nodes, subresource=proxy)

I put all the steps from the tutorial in a bash script, which is in a repository I created here.

The complete logs from running my script is below:

😄  minikube v1.26.0 on Darwin 12.4
✨  Automatically selected the docker driver. Other choices: virtualbox, ssh
📌  Using Docker Desktop driver with root privileges
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
💾  Downloading Kubernetes v1.24.1 preload ...
    > preloaded-images-k8s-v18-v1...: 405.83 MiB / 405.83 MiB  100.00% 22.87 Mi
🔥  Creating docker container (CPUs=2, Memory=8100MB) ...
🌐  Found network options:
    ▪ no_proxy=,.docker.internal
🐳  Preparing Kubernetes v1.24.1 on Docker 20.10.17 ...
    ▪ env NO_PROXY=,.docker.internal
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
💡  After the addon is enabled, please run "minikube tunnel" and your ingress resources would be available at "127.0.0.1"
    ▪ Using image k8s.gcr.io/ingress-nginx/controller:v1.2.1
    ▪ Using image k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.1.1
    ▪ Using image k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.1.1
🔎  Verifying ingress addon...
🌟  The 'ingress' addon is enabled
namespace/opa created
Context "opa-tutorial" created.
Switched to context "opa-tutorial".
Generating RSA private key, 2048 bit long modulus
...................+++
......+++
e is 65537 (0x10001)
Generating RSA private key, 2048 bit long modulus
.......................+++
....................................................+++
e is 65537 (0x10001)
Signature ok
subject=/CN=opa.opa.svc
Getting CA Private Key
secret/opa-server created
+ docker run --rm --name bundle-server -d -p 8888:80 -v /Users/Impakt/projects/opa/policies:/usr/share/nginx/html:ro nginx:latest
cc02d7c6e08f9040ff6eacbc29513696a43d979e1a6ea071f9b29ab187b99483
+ set +x
clusterrolebinding.rbac.authorization.k8s.io/opa-viewer created
role.rbac.authorization.k8s.io/configmap-modifier created
rolebinding.rbac.authorization.k8s.io/opa-configmap-modifier created
service/opa created
deployment.apps/opa created
namespace/kube-system labeled
namespace/opa labeled
validatingwebhookconfiguration.admissionregistration.k8s.io/opa-validating-webhook created
The OPA logs will now appear, press ctrl+c to exit the logs and continue the script. If the container is still starting, enter \e[32mkubectl logs -l app=opa -c opa -f\e[0m to try again until it's up
That's \e[32mkubectl logs -l app=opa -c opa -f\e[0m
Error from server (InternalError): Internal error occurred: Authorization error (user=kube-apiserver-kubelet-client, verb=get, resource=nodes, subresource=proxy)

Steps To Reproduce

  1. Pull the script and config from my repository
  2. Run ./1-install.sh

Expected behavior

OPA starts without any auth/ssl issues

Additional context

  • I also tried changing the services.default.url in the admission-controller.yaml file to http://host.docker.internal:8888 and I'm still getting Authorization errors.
  • Running minikube with the virtualbox driver doesn't work either (same errors)
  • I'm running this on macOS 12.4
@anderseknert
Copy link
Member

The kubernetes API server logs tell us why the certificate is considered invalid:

I0719 10:41:58.891272       1 server_cert_deprecations.go:68] insecure-sha1.invalid-cert.kubernetes.io: invalid certificate detected connecting to "opa.opa.svc": uses an insecure SHA-1 signature
W0719 10:41:58.891317       1 dispatcher.go:153] Failed calling webhook, failing closed validating-webhook.openpolicyagent.org: failed calling webhook "validating-webhook.openpolicyagent.org": failed to call webhook: Post "https://opa.opa.svc:443/?timeout=10s": x509: certificate signed by unknown authority (possibly because of "x509: cannot verify signature: insecure algorithm SHA1-RSA (temporarily override with GODEBUG=x509sha1=1)" while trying to verify candidate authority certificate "admission_ca")

This can be fixed by adding e.g. -sha256 to the openssl commands where hashing is used. This however uncovers the next problem with the current configuration:

I0719 11:10:28.396069       1 server_cert_deprecations.go:68] missing-san.invalid-cert.kubernetes.io: invalid certificate detected connecting to "opa.opa.svc": relies on a legacy Common Name field instead of the SAN extension for subject validation
W0719 11:10:28.396108       1 dispatcher.go:153] Failed calling webhook, failing closed validating-webhook.openpolicyagent.org: failed calling webhook "validating-webhook.openpolicyagent.org": failed to call webhook: Post "https://opa.opa.svc:443/?timeout=10s": x509: certificate relies on legacy Common Name field, use SANs instead

We'll need to update the config to use the SAN extension name rather than the common name.

anderseknert added a commit to anderseknert/opa that referenced this issue Jul 19, 2022
Two unrelated issues would break the Kubernetes tutorial in
recent kube versions. The first one being the SHA1 hash used
by default in at least older versions of OpenSSL, which is no
longer accepted by Kubernetes. Easy fix.

The next one is definitely a head scratcher - for whatever
reason, the subjectAltName previously provided in the config
didn't seem to be picked up in certificate signing requests.
Older versions of Kubernetes - or Go, really - would accept
the common name (CN), but more recent ones require the use of
subjectAltName, so it's possible this never "worked" as intended
but was ignored as the CN was used instead.

The docs on the topic however all suggest that the previous
config _should_ have worked, and after having spent a long time
trying to figure out why it didn't, I've found nothing to
provide any insights here. Best I have is "works on my
machine", so if anyone else would want to try this out to make
sureit works on theirs too, that'd be great.

* Use explicit hashing algorithm
* Specify -extensions as this does not seem to be picked up when
  provided in config only.

Fixes open-policy-agent#4902

Signed-off-by: Anders Eknert <anders@eknert.com>
anderseknert added a commit that referenced this issue Jul 19, 2022
Two unrelated issues would break the Kubernetes tutorial in
recent kube versions. The first one being the SHA1 hash used
by default in at least older versions of OpenSSL, which is no
longer accepted by Kubernetes. Easy fix.

The next one is definitely a head scratcher - for whatever
reason, the subjectAltName previously provided in the config
didn't seem to be picked up in certificate signing requests.
Older versions of Kubernetes - or Go, really - would accept
the common name (CN), but more recent ones require the use of
subjectAltName, so it's possible this never "worked" as intended
but was ignored as the CN was used instead.

The docs on the topic however all suggest that the previous
config _should_ have worked, and after having spent a long time
trying to figure out why it didn't, I've found nothing to
provide any insights here. Best I have is "works on my
machine", so if anyone else would want to try this out to make
sureit works on theirs too, that'd be great.

* Use explicit hashing algorithm
* Specify -extensions as this does not seem to be picked up when
  provided in config only.

Fixes #4902

Signed-off-by: Anders Eknert <anders@eknert.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants