Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dashboard invalid certificate #1046

Closed
toshovski opened this issue Mar 20, 2020 · 31 comments
Closed

Dashboard invalid certificate #1046

toshovski opened this issue Mar 20, 2020 · 31 comments
Labels

Comments

@toshovski
Copy link

When I execute microk8s.enable dashboard, is there a way to pass certificates? The current certificate is invalid and chrome doesn't allow me to access the dashboard.

I get the following error, and I cannot accept the risks anymore. On firefox I can still access it.

Your connection is not private
Attackers might be trying to steal your information from strzyga.abasag.intra (for example, passwords, messages, or credit cards). Learn more
NET::ERR_CERT_INVALID

@ktsakalozos
Copy link
Member

Hi @toshovski

Chrome blocks self signed certificates. This might help you: #945 (comment)

@toshovski
Copy link
Author

toshovski commented Mar 27, 2020

Not all of them. the dashboard is signing a bad certificates. Is there a way to pass my own certificate by using micrk8s.enable dashboard?

This is how it looks like:

The certificate:
image

The page:
image

When I add a CN to the certificate, Google Chrome still allows me to proceed.

The certificate:
image

The page:
image

I could disable the certificate check as a workaround for now, but this is not a solution

@bzamecnik
Copy link

Same here (microk8s 1.18), dashboard (v2.0.0-rc5) certificate is invalid even if I curl to the service locally. How did you manage to fill the common name and generate a new certificate?

$ kubectl -n kube-system describe service kubernetes-dashboard|grep Endpoints
Endpoints:         10.1.88.29:8443

curl https://10.1.88.29:8443
curl: (60) server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none

$ curl -kv https://10.1.88.29:8443                                                                         
* Rebuilt URL to: https://10.1.88.29:8443/
*   Trying 10.1.88.29...
* Connected to 10.1.88.29 (10.1.88.29) port 8443 (#0)
* found 148 certificates in /etc/ssl/certs/ca-certificates.crt
* found 603 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_ECDSA_AES_128_GCM_SHA256
*        server certificate verification SKIPPED
*        server certificate status verification SKIPPED
* error fetching CN from cert:The requested data were not available.
*        common name:  (does not match '10.1.88.29')
*        server certificate expiration date OK
*        server certificate activation date OK
*        certificate public key: EC
*        certificate version: #3
*        subject: 
*        start date: Tue, 21 Apr 2020 11:11:09 GMT
*        expire date: Wed, 21 Apr 2021 11:11:09 GMT
*        issuer: 
*        compression: NULL

@bzamecnik
Copy link

The problem is that the self-signed certificate does not use the subjectAltName. Following the advices in https://stackoverflow.com/a/43666288/329263 I was able to generate a certificate with all the needed hostnames/IPs (cluster ip, localhost, LAN IP/name), replace the secret kubernetes-dashboard-certs and specify it into the kubernetes-dashboard configuration as advised in https://github.com/kubernetes/dashboard/blob/master/docs/user/certificate-management.md#self-signed-certificate. Finally I'm able to access the dashboard via the kube proxy.

./create_root_cert_and_key.sh
./create_certificate_for_domain.sh <LAN_HOSTNAME> <LAN_IP> localhost 127.0.0.1 <CLUSTER_IP>
mv <LAN_HOSTNAME>.crt dashboard.crt

kubectl -n kube-system delete secret kubernetes-dashboard-certs
kubectl -n kube-system create secret generic kubernetes-dashboard-certs --from-file=dashboard.crt --from-file=device.key
kubectl -n kube-system edit deploy kubernetes-dashboard -o yaml

# add:
- args:
  - --tls-cert-file=/dashboard.crt
  - --tls-key-file=/device.key

openssl s_client -showcerts -connect <CLUSTER_IP>:443

kubectl port-forward -n kube-system service/kubernetes-dashboard 10443:443 --address 0.0.0.0

desktop$ open https://<LAN_HOSTNAME>

@hemna
Copy link

hemna commented Jun 12, 2020

I am having this same problem. I had to use an old version of firefox to access the dashboard. I even added the cert to my macos keychain access and said trust, and chrome still didn't like it.

I tried running the scripts in comment #1046 (comment), but it didn't generate a dashboard.crt, so I got stuck.

@Andras-Csanyi
Copy link

I installed microk8s yesterday, latest/edge channel, and I got the same certification error.
Changing Chrome to accept self-signed certificates doesn't help. Based on what I read in the #945 and in the PR #970 I still need to adjust certificates on the machine kubernetes is running.
For me, who doesn't like to deal with certificates it is a bad user experience facing with this especially with the promise of Zero-ops kubernetes.

@demanLiu
Copy link

demanLiu commented Aug 30, 2020

kubernetes/dashboard#2995 (comment)
This worked
namespace may be different (kube-system)

@ruudboon
Copy link

ruudboon commented Sep 3, 2020

This is blocking me as well

@ktsakalozos
Copy link
Member

@numman-ali
Copy link

Chrome Quick-Fix:
Go to chrome://flags/#allow-insecure-localhost
And enable "Allow invalid certificates for resources loaded from localhost."

@hemna
Copy link

hemna commented Nov 26, 2020

Just installed microk8s dashboard on the latest snap --classic microk8s channel. Have the same problem. The dashboard cert is invalid and can't get any browser to use it. This needs to get fixed.

@ghandmann
Copy link

Same problem here. Installed microk8s from the edge-channel. Started the dashboard and cannot access it via Chrome. Even the Workaround with Allow invalid certificates for resources loaded from localhost. is not allowing me to access the dashboard.

Here is a quick dump of the presented certificate made with the openssl s_client:

openssl s_client -connect 10.152.183.176:443
CONNECTED(00000005)
depth=0 
verify error:num=18:self signed certificate
verify return:1
depth=0 
verify return:1
---
Certificate chain
 0 s:
   i:
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIBADCBpqADAgECAhAXt3NvfFYj0jLAsyyR1Mf5MAoGCCqGSM49BAMCMAAwHhcN
MjAxMTI3MjEyMjU2WhcNMjExMTI3MjEyMjU2WjAAMFkwEwYHKoZIzj0CAQYIKoZI
zj0DAQcDQgAE+8efMOScjbJJdhofFE5JMcS8DWutAgaA/+OP1wbdt9WtlKH1ovYu
LjnYgM8KD4kuN1HbZ9avnYkiA2QvPmNpZaMCMAAwCgYIKoZIzj0EAwIDSQAwRgIh
ANyrJrxlOX3S/7cigOwCwCEj1cxFgvP+F5KD/MuXVYyKAiEA7Zr17umwbvfCmhtp
2M76xSHvI7FuSkWp+VDao41JwF4=
-----END CERTIFICATE-----
subject=

issuer=

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: ECDSA
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 615 bytes and written 380 bytes
Verification error: self signed certificate

So beside that the Cert is self signed, it is virtually empty? No subject, no issuer. At least it is valid for one year. :D

@midnightcodr
Copy link

midnightcodr commented Dec 7, 2020

My microk8s was installed using --classic and here's what works for me:

mkdir certs
openssl req -nodes -newkey rsa:2048 -keyout certs/dashboard.key -out certs/dashboard.csr -subj "/C=/ST=/L=/O=/OU=/CN=*"
openssl x509 -req -sha256 -days 3650 -in certs/dashboard.csr -signkey certs/dashboard.key -out certs/dashboard.crt

kubectl -n kube-system delete secret kubernetes-dashboard-certs
kubectl -n kube-system create secret generic kubernetes-dashboard-certs --from-file=dashboard.crt --from-file=dashboard.key
kubectl -n kube-system edit deploy kubernetes-dashboard -o yaml

# modify section args as follows
          args:
            - --tls-cert-file=/dashboard.crt
            - --tls-key-file=/dashboard.key
            #- --auto-generate-certificates

@gregtoth
Copy link

My microk8s was installed using --classic and here's what works for me:

mkdir certs
openssl req -nodes -newkey rsa:2048 -keyout certs/dashboard.key -out certs/dashboard.csr -subj "/C=/ST=/L=/O=/OU=/CN=*"
openssl x509 -req -sha256 -days 3650 -in certs/dashboard.csr -signkey certs/dashboard.key -out certs/dashboard.crt

kubectl -n kube-system delete secret kubernetes-dashboard-certs
kubectl -n kube-system create secret generic kubernetes-dashboard-certs --from-file=dashboard.crt --from-file=dashboard.key
kubectl -n kube-system edit deploy kubernetes-dashboard -o yaml

# modify section args as follows
          args:
            - --tls-cert-file=/dashboard.crt
            - --tls-key-file=/dashboard.key
            #- --auto-generate-certificates

This worked for me with some minor mods:

openssl req -nodes -newkey rsa:2048 -keyout dashboard.key -out dashboard.csr -subj "/C=/ST=/L=/O=/OU=/CN=*"
openssl x509 -req -sha256 -days 3650 -in dashboard.csr -signkey dashboard.key -out dashboard.crt

kubectl -n kube-system delete secret kubernetes-dashboard-certs
kubectl -n kube-system create secret generic kubernetes-dashboard-certs --from-file=dashboard.crt --from-file=dashboard.key
kubectl -n kube-system edit deploy kubernetes-dashboard -o yaml

# modify section args as follows
          args:
            - --tls-cert-file=dashboard.crt
            - --tls-key-file=dashboard.key
            #- --auto-generate-certificates

After stopping and restarting microk8s dashboard-proxy I can now log in using Chrome 88.

@cottrell
Copy link

mkdir certs
openssl req -nodes -newkey rsa:2048 -keyout certs/dashboard.key -out certs/dashboard.csr -subj "/C=/ST=/L=/O=/OU=/CN=*"
openssl x509 -req -sha256 -days 3650 -in certs/dashboard.csr -signkey certs/dashboard.key -out certs/dashboard.crt

kubectl -n kube-system delete secret kubernetes-dashboard-certs
kubectl -n kube-system create secret generic kubernetes-dashboard-certs --from-file=dashboard.crt --from-file=dashboard.key
kubectl -n kube-system edit deploy kubernetes-dashboard -o yaml

What does the "modify section args" part mean? I'm not understanding what this is applied to.

@cottrell
Copy link

mkdir certs
openssl req -nodes -newkey rsa:2048 -keyout certs/dashboard.key -out certs/dashboard.csr -subj "/C=/ST=/L=/O=/OU=/CN=*"
openssl x509 -req -sha256 -days 3650 -in certs/dashboard.csr -signkey certs/dashboard.key -out certs/dashboard.crt

kubectl -n kube-system delete secret kubernetes-dashboard-certs
kubectl -n kube-system create secret generic kubernetes-dashboard-certs --from-file=dashboard.crt --from-file=dashboard.key
kubectl -n kube-system edit deploy kubernetes-dashboard -o yaml

What does the "modify section args" part mean? I'm not understanding what this is applied to.

Do you mean this?

        sudo microk8s dashboard-proxy --address 0.0.0.0 --tls-cert-file=dashboard.crt --tls-key-file=dashboard.key

@gregtoth
Copy link

Run these commands:

openssl req -nodes -newkey rsa:2048 -keyout dashboard.key -out dashboard.csr -subj "/C=/ST=/L=/O=/OU=/CN=*"
openssl x509 -req -sha256 -days 3650 -in dashboard.csr -signkey dashboard.key -out dashboard.crt

kubectl -n kube-system delete secret kubernetes-dashboard-certs
kubectl -n kube-system create secret generic kubernetes-dashboard-certs --from-file=dashboard.crt --from-file=dashboard.key
kubectl -n kube-system edit deploy kubernetes-dashboard -o yaml

The last command opens a yaml configuration in editor (e.g. vi). Look for the 'args' section and change it so it looks like this:

          args:
            - --tls-cert-file=dashboard.crt
            - --tls-key-file=dashboard.key
            #- --auto-generate-certificates

This comments out the auto-generate-certificates option and explicitly sets the key and cert file.

Then save and exit the editor to apply the changes.

If 'microk8s dashboard-proxy' is already running, press Control-C to stop it.

Then restart it with this command:

microk8s dashboard-proxy

@cottrell
Copy link

cottrell commented Jan 26, 2021

kubectl -n kube-system create secret generic kubernetes-dashboard-certs --from-file=dashboard.crt --from-file=dashboard.key

Interesting. I'm getting timeouts on the second to last command. Trying to see how to get more information about the timeout details.

kubectl -n kube-system create secret generic kubernetes-dashboard-certs --from-file=dashboard.crt --from-file=dashboard.key

Unable to connect to the server: dial tcp 192.168.99.100:8443: i/o timeout

And the delete just hangs too (I didn't wait long enough for it to finish).

@cottrell
Copy link

kubectl -n kube-system create secret generic kubernetes-dashboard-certs --from-file=dashboard.crt --from-file=dashboard.key

Interesting. I'm getting timeouts on the second to last command. Trying to see how to get more information about the timeout details.

kubectl -n kube-system create secret generic kubernetes-dashboard-certs --from-file=dashboard.crt --from-file=dashboard.key

Unable to connect to the server: dial tcp 192.168.99.100:8443: i/o timeout

And the delete just hangs too (I didn't wait long enough for it to finish).

So the only trick is to add microk8s in front of those commands. Seems obvious once you know.


        microk8s kubectl -n kube-system delete secret kubernetes-dashboard-certs
        microk8s kubectl -n kube-system create secret generic kubernetes-dashboard-certs --from-file=dashboard.crt --from-file=dashboard.key
        microk8s kubectl -n kube-system edit deploy kubernetes-dashboard -o yaml

@mikabytes
Copy link

I managed to get this working after several gotchas...

Complete steps:

  1. Make sure dashboard-proxy is not running

  2. Run these commands:

mkdir /tmp/certs
cd /tmp/certs

openssl req -nodes -newkey rsa:2048 -keyout dashboard.key -out dashboard.csr -subj "/C=/ST=/L=/O=/OU=/CN=*"
openssl x509 -req -sha256 -days 3650 -in dashboard.csr -signkey dashboard.key -out dashboard.crt

sudo microk8s kubectl -n kube-system delete secret kubernetes-dashboard-certs
sudo microk8s kubectl -n kube-system create secret generic kubernetes-dashboard-certs --from-file=dashboard.crt --from-file=dashboard.key
sudo microk8s kubectl -n kube-system edit deploy kubernetes-dashboard -o yaml
  1. In the editor that shows up replace spec.template.spec.containers.args:
....

    spec:
      containers:
      - args:
        - --tls-cert-file=dashboard.crt
        - --tls-key-file=dashboard.key
        - --namespace=kube-system

....
  1. Start dashboard-proxy

  2. In Chrome 88, open console and long-press the reload button until a menu shows. Pick "Empty Cache and Hard Reload":

image

  1. Then, you'll get a new error. Click Advanced, and then "Proceed, unsafe":

image

@ljtill
Copy link

ljtill commented Aug 13, 2021

Another way I've just discovered to navigate around this issue (Tested this in Edge Chromium, but it should work in Chrome too)... If you click anywhere in the window and type 'thisisunsafe' it'll reload the page and navigate to the Kubernetes Dashboard login...

@mikabytes
Copy link

Another way I've just discovered to navigate around this issue (Tested this in Edge Chromium, but it should work in Chrome too)... If you click anywhere in the window and type 'thisisunsafe' it'll reload the page and navigate to the Kubernetes Dashboard login...

Yep, that will work in all Chromium-based browsers. But neither of these workarounds are ideal. I'm wondering, how come this isn't getting more attention? The bug has been up for well over a year. microk8s is such a great project, and it's disheartening for people who are just getting started to hit this kind of wall so early while following tutorials online.

@ltvan
Copy link

ltvan commented Dec 13, 2021

The code to generate self-signed certificate above missing required extensions so that the browser see that the certificate is invalid (not valid for signing a server). For some people, it may happen to be valid because the openssl.cnf already contained predefined configuration which allow to create a valid SSL certificate.

To be sure, you can use these commands to generate a valid self-signed certificate:

mkdir ~/certs
cd ~/certs

tee openssl.cnf <<EOF
[ req ]
default_bits        = 2048
distinguished_name  = req_distinguished_name
string_mask         = utf8only
default_md          = sha256
x509_extensions     = server_cert
prompt			    = no

[ req_distinguished_name ]
commonName          = localhost

[ server_cert ]
basicConstraints = CA:FALSE
nsCertType = server
nsComment = "OpenSSL Generated Server Certificate"
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer:always
keyUsage = critical, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names

[alt_names]
DNS.1 = localhost
DNS.2 = *
EOF

# generate both key and certificate in one step
openssl req -nodes -newkey rsa:2048 -keyout dashboard.key -new -x509 -days 3650 -out dashboard.crt -extensions server_cert -config openssl.cnf

Then update the certificate to k8s dashboard

microk8s kubectl -n kube-system delete secret kubernetes-dashboard-certs
microk8s kubectl -n kube-system create secret generic kubernetes-dashboard-certs --from-file=dashboard.crt --from-file=dashboard.key

Update the dashboard config to use these certs:

microk8s kubectl -n kube-system edit deploy kubernetes-dashboard -o yaml

# edit spec.template.spec.containers.args as
....
    spec:
      containers:
      - args:
        - --tls-cert-file=dashboard.crt
        - --tls-key-file=dashboard.key
        - --namespace=kube-system
....

Restart the dashboard pod so that it recognizes the new certificate. In my case, the namespace contains dashboard pod is kube-system. It may be different in your case:

# look and see which namespace your pod is in
microk8s kubectl get pod --all-namespaces

# delete all dashboard pods from kube-system
microk8s kubectl delete -n kube-system $(microk8s kubectl get pod --all-namespaces -o name | grep dashboard)

Start the dashboard proxy again

microk8s dashboard-proxy

Now you will be able to see the bypass link when opening the dashboard in browser.

@planetf1
Copy link

planetf1 commented Jan 6, 2022

I'm writing a tutorial for our project, which will use microk8s, and hit this when documenting the dashboard for those new users to understand what is running. (I am almost entirely CLI .... )

Unfortunately this issue is a real impediment for the new user experience. I do empathise with the pain of creating self-signed certs. I struggled myself with it on egeria. I ended up with these -> https://github.com/odpi/egeria/tree/master/open-metadata-resources/open-metadata-deployment/certificates with a ref from https://odpi.github.io/egeria-docs/guides/admin/omag-server-platform-transport-level-security/#example-script-to-launch-egeria

I'm hitting the error in microk8s using safari on Monterey. There is no user bypass. As mentioned above, some of the certs need additional settings to allow safari/chrome to work at all. Of course this is only ever for demos, tutorials

planetf1 added a commit to planetf1/egeria-docs that referenced this issue Jan 6, 2022
…ue to canonical/microk8s#1046

Signed-off-by: Nigel Jones <nigel.l.jones+git@gmail.com>
@olahouze
Copy link

olahouze commented Mar 1, 2022

The code to generate self-signed certificate above missing required extensions so that the browser see that the certificate is invalid (not valid for signing a server). For some people, it may happen to be valid because the openssl.cnf already contained predefined configuration which allow to create a valid SSL certificate.

To be sure, you can use these commands to generate a valid self-signed certificate:

mkdir ~/certs
cd ~/certs

tee openssl.cnf <<EOF
[ req ]
default_bits        = 2048
distinguished_name  = req_distinguished_name
string_mask         = utf8only
default_md          = sha256
x509_extensions     = server_cert
prompt			    = no

[ req_distinguished_name ]
commonName          = localhost

[ server_cert ]
basicConstraints = CA:FALSE
nsCertType = server
nsComment = "OpenSSL Generated Server Certificate"
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer:always
keyUsage = critical, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names

[alt_names]
DNS.1 = localhost
DNS.2 = *
EOF

# generate both key and certificate in one step
openssl req -nodes -newkey rsa:2048 -keyout dashboard.key -new -x509 -days 3650 -out dashboard.crt -extensions server_cert -config openssl.cnf

Then update the certificate to k8s dashboard

microk8s kubectl -n kube-system delete secret kubernetes-dashboard-certs
microk8s kubectl -n kube-system create secret generic kubernetes-dashboard-certs --from-file=dashboard.crt --from-file=dashboard.key

Update the dashboard config to use these certs:

microk8s kubectl -n kube-system edit deploy kubernetes-dashboard -o yaml

# edit spec.template.spec.containers.args as
....
    spec:
      containers:
      - args:
        - --tls-cert-file=dashboard.crt
        - --tls-key-file=dashboard.key
        - --namespace=kube-system
....

Restart the dashboard pod so that it recognizes the new certificate. In my case, the namespace contains dashboard pod is kube-system. It may be different in your case:

# look and see which namespace your pod is in
microk8s kubectl get pod --all-namespaces

# delete all dashboard pods from kube-system
microk8s kubectl delete -n kube-system $(microk8s kubectl get pod --all-namespaces -o name | grep dashboard)

Start the dashboard proxy again

microk8s dashboard-proxy

Now you will be able to see the bypass link when opening the dashboard in browser.

This solution work for me.

Could you implement this on new version of microk8s ?

Best regards

@zoepfchen
Copy link

I've found this solution to renew the server certificates. works for me. after renewing all nodes are back alive: https://www.etissimo.de/blog/2022/04/03/renew-microk8s-kubernetes-server-certificates-annually/?lang=en

@olahouze
Copy link

olahouze commented Apr 4, 2022

It would be easier if this feature was native in the product... right?

@zoepfchen
Copy link

yes. it should be integrated into the product, also to avoid unpleasant system failures.

@soulteary
Copy link

I made a simple and small (~3MB) docker tool to generate a self-signed certs. It is relatively simple to add multiple DNS domain names or specify good-looking basic information.

I hope it will help friends who need to make self-signed (generate a certificate that can be used by k8s):

docker run --rm -it -v `pwd`/certs:/ssl soulteary/certs-maker --FOR_K8S=on

If you need to add multiple domain names, just run:

docker run --rm -it -v `pwd`/certs:/ssl soulteary/certs-maker --FOR_K8S=on --CERT_DNS=apple.com,orange.com,pear.com

For more parameters and usage, you can refer to the project and code: https://github.com/soulteary/certs-maker

@stale
Copy link

stale bot commented Sep 10, 2023

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the inactive label Sep 10, 2023
@stale stale bot closed this as completed Oct 10, 2023
@mikabytes
Copy link

I'm afraid I have to disagree with this being closed as the community obviously thinks it's relevant.

The best would be if a certificate were generated automatically by microk8s as part of enabling the dashboard.

I would find it acceptable to update the documentation to help new users deal with the limitations.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests