--tls-verify fails with "cannot validate certificate for 127.0.0.1"

[scjody]

Output of helm version:

Client: &version.Version{SemVer:"v2.9.1", GitCommit:"20adb27c7c5868466912eebdf6664e7390ebe710", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.9.1", GitCommit:"20adb27c7c5868466912eebdf6664e7390ebe710", GitTreeState:"clean"}

Output of kubectl version:

Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.0", GitCommit:"fc32d2f3698e36b93322a3465f63a14e9f0eaead", GitTreeState:"clean", BuildDate:"2018-03-26T16:55:54Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"9+", GitVersion:"v1.9.6-gke.1", GitCommit:"cb151369f60073317da686a6ce7de36abe2bda8d", GitTreeState:"clean", BuildDate:"2018-04-07T22:06:59Z", GoVersion:"go1.9.3b4", Compiler:"gc", Platform:"linux/amd64"}

Cloud Provider/Platform (AKS, GKE, Minikube etc.): GKE

Problem encountered

I configured Tiller with TLS following the instructions at https://docs.helm.sh/tiller_ssl/ .

When I run helm ls --tls --tls-verify, a certificate validation error is returned instead of the expected list:

$ helm ls --tls --tls-verify
Error: x509: cannot validate certificate for 127.0.0.1 because it doesn't contain any IP SANs

Certificate information

$ openssl x509 -noout -text -in tiller-stage.cert.pem
Certificate:
    Data:
        Version: 1 (0x0)
        Serial Number: 13067169092421557806 (0xb557e90e85c01e2e)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=CA, ST=Quebec, L=Montreal, O=Plotly, Inc., OU=devops, CN=tiller/emailAddress=jody@plot.ly
        Validity
            Not Before: May 30 19:35:20 2018 GMT
            Not After : May 30 19:35:20 2019 GMT
        Subject: C=CA, ST=Quebec, L=Montreal, O=Plotly, Inc., OU=devops, CN=tiller stage
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (4096 bit)
                Modulus:
                    00:e3:62:96:57:37:7f:66:a7:4b:29:ba:20:4f:ed:
                    dc:3e:b7:f8:28:11:9e:03:d0:32:0e:a4:e9:3a:cb:
                    e5:56:16:89:2f:43:02:c3:40:81:38:50:79:1e:83:
                    ed:53:4a:50:ea:bf:af:69:b6:71:38:3a:8f:df:b2:
                    b2:8f:63:71:b2:b3:98:43:60:ed:05:66:d4:4e:99:
                    ad:b6:b1:bf:53:70:c4:e5:e9:b2:37:98:07:75:5b:
                    e8:19:33:d2:a5:d4:34:0e:0e:90:36:52:03:33:74:
                    c3:47:0c:18:be:c3:b3:fe:29:3f:00:5d:56:9e:3f:
                    be:02:c8:d1:b2:9d:27:4f:be:81:e4:06:dd:38:94:
                    3a:df:a1:65:54:99:3d:a2:aa:bf:e3:06:7f:d1:0e:
                    d8:77:7e:c8:3d:9c:16:e3:64:c9:25:35:44:90:9b:
                    49:12:26:30:2f:32:a9:d7:93:3c:00:ea:c5:9e:2c:
                    c9:f9:11:60:05:d8:a2:99:d5:5c:e6:50:91:31:5c:
                    b9:c5:79:1a:57:17:b3:51:94:36:c7:93:9a:69:d8:
                    27:e3:37:5a:a9:71:2e:d7:e2:cf:68:d8:3f:c3:a3:
                    93:9a:cc:99:40:7c:dd:f8:97:13:39:69:ff:bc:d3:
                    c0:83:77:36:f4:76:05:3c:99:ec:6e:9e:9b:3c:fa:
                    03:8b:08:9a:70:4b:3f:a3:13:02:c5:0f:ea:a6:c5:
                    cf:0e:70:b0:61:6c:70:cb:36:fa:c7:92:60:b0:18:
                    04:1e:f0:9f:c8:78:6f:ff:05:2b:f1:cb:28:4f:8f:
                    ce:84:cb:55:2d:8f:1c:82:45:a5:b2:25:71:82:9c:
                    eb:9a:e3:a2:85:81:20:bf:87:dc:57:8a:c6:32:5c:
                    11:5d:89:3c:ce:24:36:64:87:44:89:36:82:4f:52:
                    78:ad:55:ac:68:a3:d7:83:d1:8d:2e:af:65:e1:38:
                    f5:33:ce:18:03:30:e5:30:04:ba:49:c7:e3:06:28:
                    7e:08:5a:5c:b3:45:22:28:85:ed:1d:93:d4:76:30:
                    0a:8f:de:7a:0d:ec:ff:73:c1:67:2d:31:97:77:5f:
                    92:52:e9:f8:20:00:e0:cf:e4:b3:51:79:42:2c:f2:
                    bb:6f:b8:f8:84:06:7f:6a:51:da:62:1b:84:f1:ca:
                    1e:14:f1:f4:fc:8f:aa:1c:d9:34:31:2f:a1:10:e0:
                    2d:31:e7:20:8c:f1:ca:19:21:fa:9d:b7:20:bf:0a:
                    79:0c:a9:62:2e:ac:5b:c6:ef:9b:9f:cb:15:99:27:
                    17:dd:6c:38:0e:bd:fe:86:b9:ed:b8:03:86:b8:e3:
                    52:65:a7:a1:77:23:a8:d8:fc:b6:10:09:b4:eb:3b:
                    55:db:6f
                Exponent: 65537 (0x10001)
    Signature Algorithm: sha256WithRSAEncryption
         6b:ff:c7:b9:a5:85:89:cb:97:3c:3b:1b:90:46:11:12:35:c0:
         9d:3e:34:7a:a4:04:a6:4e:c5:c2:c1:28:40:db:9d:d5:fc:73:
         54:4b:1d:3d:1e:cc:da:e3:bc:a1:46:a5:08:98:62:67:9c:aa:
         9f:41:e8:30:7c:5c:68:e0:89:a7:67:52:5c:19:a4:41:6c:54:
         56:b5:df:fe:23:bc:e7:99:92:15:22:6c:1a:74:c3:c6:54:69:
         1f:90:22:b1:b0:7e:6c:8a:fd:e1:eb:7c:80:d0:3d:f5:85:bc:
         ae:d9:6f:9a:7b:6d:06:a4:67:8d:42:1b:9b:67:0b:26:38:5b:
         63:9a:1c:70:aa:6d:a5:50:0d:7d:c9:e6:c3:0d:51:5a:0f:94:
         7f:ef:ec:6a:3f:8a:e6:66:5a:ca:04:c8:fc:02:70:60:33:52:
         bb:9a:32:e7:e0:10:9d:5d:51:6a:e9:29:26:c7:78:26:f8:69:
         62:ad:b8:f5:75:b7:e9:18:64:0d:a9:58:4c:06:10:c3:03:6f:
         35:49:03:a7:88:05:27:92:7b:b4:01:1b:9d:b5:bb:7a:eb:e8:
         27:20:04:4e:07:29:0d:51:85:e9:08:1f:33:c5:3e:b1:28:df:
         4f:c6:d9:f6:61:fb:6b:25:69:bb:8a:7a:d7:3a:8b:a4:d9:ab:
         20:c6:f6:ae:da:66:ea:3e:a7:f5:1c:1d:1f:46:b9:3c:38:86:
         7f:f7:96:e0:77:e1:39:66:de:46:10:8f:39:b1:df:c4:f7:ef:
         cf:7a:7b:db:29:19:2c:ce:35:6c:1b:50:da:c4:91:24:6c:1f:
         13:06:ec:6a:8c:9d:1e:3c:e5:47:2c:ff:ce:ec:55:8d:62:63:
         0d:fd:8e:b2:19:16:7b:29:ec:a3:4a:84:af:66:3a:85:e5:0e:
         b1:df:81:18:f6:23:21:da:a8:e5:c6:d2:a4:85:80:cb:32:61:
         07:02:e6:24:60:82:e9:37:03:c9:ed:f6:fc:f7:36:9a:6c:76:
         04:b2:ea:c2:99:4b:ee:99:d2:69:c1:4a:30:39:70:fd:6c:38:
         bc:af:04:82:58:0b:25:43:2b:dc:c3:58:e5:41:16:5d:0c:f7:
         da:93:3e:31:7d:ff:fb:d4:44:38:e1:91:1c:6f:a1:aa:14:f4:
         2d:a7:2d:52:46:8a:e6:87:3c:61:92:bd:16:56:05:e5:8d:cf:
         0d:66:aa:f4:47:f6:61:1e:65:a3:a2:41:ee:12:ea:f2:f9:5d:
         8a:f7:b4:d9:67:de:41:73:3b:16:37:dc:8a:50:79:6d:82:a3:
         74:07:83:e4:f1:d7:a2:4d:e2:07:1b:1e:0a:22:dd:da:5a:35:
         f5:c7:10:4d:e8:c2:f9:54

$ openssl x509 -noout -text -in ca.cert.pem
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 10910624137687925762 (0x976a52d8a4b10402)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=CA, ST=Quebec, L=Montreal, O=Plotly, Inc., OU=devops, CN=tiller/emailAddress=jody@plot.ly
        Validity
            Not Before: Apr 26 19:45:54 2018 GMT
            Not After : Apr 21 19:45:54 2038 GMT
        Subject: C=CA, ST=Quebec, L=Montreal, O=Plotly, Inc., OU=devops, CN=tiller/emailAddress=jody@plot.ly
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (4096 bit)
                Modulus:
                    00:c3:9a:d0:d8:6f:85:b8:1b:d5:80:1e:e5:11:fe:
                    9a:82:e3:95:25:fa:ab:d6:cc:4a:93:ce:68:60:d6:
                    3a:04:7b:01:51:c6:ef:fc:38:8f:2c:07:a9:23:a9:
                    51:7e:bc:ac:16:77:1d:5b:09:e0:14:3f:62:81:d5:
                    32:77:14:26:73:4a:e5:2c:6a:0b:a5:10:59:37:c1:
                    32:c4:0e:31:18:25:35:8d:5e:06:d9:2f:42:ae:66:
                    a6:27:32:75:ee:1b:d2:33:40:65:a3:86:f7:8d:5f:
                    66:82:d7:44:e1:4f:2d:05:7c:a0:5e:5a:d3:c2:e5:
                    04:62:14:15:c7:d8:2a:9a:55:51:10:e3:4b:a5:e4:
                    e2:1e:fc:43:00:98:ea:65:17:a1:f5:2a:89:51:cc:
                    a9:de:07:65:5b:ec:4a:dc:75:23:1f:98:c0:1f:74:
                    20:08:aa:01:05:85:c9:8b:fa:7b:3c:7f:e6:37:59:
                    56:d9:e3:f6:08:41:0e:8d:06:36:40:17:36:f4:36:
                    f7:44:d0:aa:a9:80:67:22:29:5c:91:f3:c8:7c:19:
                    17:08:d2:f2:69:3b:0d:c9:6c:74:ec:b0:cc:69:da:
                    66:28:6b:40:0a:0a:4e:9a:ab:22:fd:7e:50:97:d4:
                    4c:cf:b8:7f:86:bd:54:f7:29:00:1a:6c:03:72:e5:
                    eb:96:a1:6b:f7:fa:4b:74:73:b7:a6:28:ef:1c:ed:
                    cc:c9:e0:ab:43:00:25:ee:f7:a8:11:c5:78:13:7f:
                    b9:06:40:fd:fb:bf:06:92:71:cc:4d:6c:22:e8:65:
                    ad:74:76:64:8a:7c:74:19:c9:4e:96:62:51:1c:db:
                    7e:15:1e:0e:70:0a:be:f5:45:bc:37:25:92:f9:a3:
                    5a:33:db:2e:cc:12:d9:b0:32:9e:4a:57:02:38:fc:
                    73:0e:02:b2:e3:98:c6:68:97:90:ce:27:6b:3b:ff:
                    5b:1c:8d:96:fd:89:04:f1:ea:11:76:96:8f:cb:78:
                    25:96:7c:7f:0b:4f:41:ab:7a:51:8d:48:08:5c:51:
                    b0:ad:1b:25:d8:c9:47:70:cc:ab:b2:2b:c5:a1:34:
                    4b:30:9d:43:c0:e2:1e:c6:44:98:de:1d:16:7f:fa:
                    86:3e:8c:b8:30:b9:07:94:40:fc:a0:a6:2f:65:c6:
                    b6:d9:84:6d:bd:48:82:2b:8b:9d:9a:1c:2e:6b:17:
                    9e:52:30:15:48:7f:d1:3d:f6:77:c5:42:8e:be:cd:
                    cf:f2:ae:8b:ef:3e:2e:18:d6:de:0a:47:5f:ec:d6:
                    0b:f4:28:5e:27:0d:2a:e8:3d:f9:81:79:c4:de:6d:
                    d2:63:a9:79:7e:41:07:01:d2:c9:44:6b:5b:7b:52:
                    c9:18:cb
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Subject Key Identifier:
                80:62:D6:39:DE:9A:75:65:86:47:3C:09:85:5D:8B:77:5F:6D:89:37
            X509v3 Authority Key Identifier:
                keyid:80:62:D6:39:DE:9A:75:65:86:47:3C:09:85:5D:8B:77:5F:6D:89:37

            X509v3 Basic Constraints:
                CA:TRUE
    Signature Algorithm: sha256WithRSAEncryption
         5c:80:6b:4f:f5:d6:65:31:e6:6d:2f:b3:68:2a:a6:b9:74:9d:
         cf:a7:45:d9:31:4a:a4:f4:46:e9:5e:a7:7b:54:d6:c2:ec:db:
         34:2c:8d:b6:44:aa:aa:84:6f:36:bd:7f:79:b9:11:5f:e5:06:
         96:bd:89:49:e4:8c:c7:ba:94:75:06:fb:a8:04:c8:f3:c3:56:
         af:c9:0e:b3:72:0b:be:1e:14:60:43:23:13:31:15:c2:20:f1:
         a7:80:57:e8:05:01:92:f9:3c:3a:34:3a:fb:11:42:9f:59:77:
         c2:fa:69:27:15:96:59:21:4a:82:c5:c0:44:8f:3e:0a:7a:93:
         3c:0d:f2:41:51:8c:84:52:37:b7:a8:cf:8d:a8:ef:ef:9d:b7:
         85:1c:53:76:58:ed:32:89:b4:f1:aa:bd:4b:8f:8e:d0:a0:bb:
         49:31:e9:d6:7e:1d:c1:58:8f:f2:13:c0:4f:87:24:20:b2:7c:
         72:b9:fb:55:e1:03:05:28:5c:f9:50:4a:dc:e1:d4:66:59:ff:
         38:6d:1a:9c:4c:04:79:6b:bf:61:13:d8:a1:1b:f4:28:3c:d3:
         76:31:26:b4:65:b0:06:43:fa:5f:48:57:99:41:b0:8f:f8:c7:
         c3:a7:3b:09:fc:b7:02:2b:31:1a:34:f7:7b:7c:c7:d6:bb:36:
         19:91:22:a5:6d:87:52:ab:cc:cb:22:b5:00:e3:92:1d:3c:68:
         c3:7b:ff:81:c1:34:53:00:57:84:10:d6:8d:c1:44:b2:31:cb:
         66:c1:4d:ea:e3:ec:4b:67:82:c4:d8:67:c9:cc:8a:2c:74:92:
         49:bd:31:21:e7:00:e6:49:88:9e:f2:74:dd:c0:18:86:0d:82:
         06:b8:f7:f0:6e:08:38:e0:13:93:50:b6:d6:65:31:ec:d2:63:
         2c:65:3c:64:b9:6f:dd:f0:9c:79:b8:0d:e9:0f:64:59:ad:4a:
         22:83:14:41:e3:b0:d2:5f:56:11:e9:9e:f8:90:b6:03:c8:91:
         79:43:b8:97:a4:b8:08:96:b5:ea:1f:15:c1:60:41:84:35:d0:
         b4:8a:f9:c6:7a:9f:99:0a:17:11:f1:ea:e3:a1:94:95:66:28:
         8c:e7:64:00:21:5b:d3:04:dc:1c:d7:25:3e:c5:de:ab:4c:ed:
         fe:59:3d:b1:41:d8:ed:30:90:88:78:1a:54:e1:50:b7:8e:1c:
         91:24:69:f0:fe:60:22:22:54:7b:a5:44:10:df:d3:b2:69:25:
         30:c6:7f:a7:4e:1c:93:30:3e:38:8f:1b:3d:6b:4f:18:10:da:
         e2:e6:de:7a:cf:c1:02:fa:11:17:3c:d7:01:63:87:52:4c:bd:
         ee:24:77:3b:76:c4:8e:69

Reproducibility

I verified that this happens on a new cluster with a new tiller, installed as follows:

helm init --tiller-tls --tiller-tls-cert ./tiller-stage.cert.pem --tiller-tls-key ./tiller-stage.key.pem --tiller-tls-verify --tls-ca-cert ca.cert.pem
kubectl create serviceaccount --namespace kube-system tiller
kubectl create clusterrolebinding tiller-cluster-rule --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
kubectl patch deploy --namespace kube-system tiller-deploy -p '{"spec":{"template":{"spec":{"serviceAccount":"tiller"}}}}'

[mattjmcnaughton]

Hi @scjody !

I think the issue is that you are attempting to connect to 127.0.0.1, but I presume your certificate is signed with a different domain name. The verification step notices this discrepancy and raises an exception.

Instead of connecting with 127.0.0.1, you could use the domain name you used when signing the certificate. I think the --host string controls this.

Although one thing I'm curious about is if SSL is still needed if you are only connecting over localhost? Hopefully that helps!

[scjody]

@mattjmcnaughton Thanks for the suggestions!

The certificate above has a CN of tiller stage, which obviously isn't a valid hostname. Based on your suggestion I tried tiller-server, which is what the docs use, and I also tried 127.0.0.1 and localhost.

I also tried telling helm to connect to localhost but that hangs: helm ls --tls-verify --host localhost. Is there another command you had in mind?

Regarding the need for TLS, Tiller is running on GKE (along with the rest of my cluster), and according to the docs the Kubernetes proxy (equivalent to kubectl proxy) is used to connect to Tiller. So it's really connecting to a remote machine via localhost. According to the docs, TLS should be configured:

Shared and production clusters – for the most part – should use Helm 2.7.2 at a minimum and configure TLS for each Tiller gRPC endpoint to ensure that within the cluster usage of gRPC endpoints is only for the properly authenticated identity for that endpoint. Doing so enables any number of Tiller instances to be deployed in any number of namespaces and yet no unauthenticated usage of any gRPC endpoint is possible. Finally, use Helm init with the --tiller-tls-verify option to install Tiller with TLS enabled and to verify remote certificates, and all other Helm commands should use the --tls option.

[ericruleman]

I also encountered this same issue when following the tls guide for helm/tiller and received the same error message: Error: x509: cannot validate certificate for 127.0.0.1 because it doesn't contain any IP SANs.

I attempted to list 127.0.0.1 as a SAN in the tiller cert, but that did not fix the issue. I also tried using the host flag, passing in 127.0.0.1, localhost, & the tiller service name as values.

Output of helm version:

Client: &version.Version{SemVer:"v2.8.2", GitCommit:"a80231648a1473929271764b920a8e346f6de844", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.8.2", GitCommit:"a80231648a1473929271764b920a8e346f6de844", GitTreeState:"clean"}

Output of kubectl version:

Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.7", GitCommit:"dd5e1a2978fd0b97d9b78e1564398aeea7e7fe92", GitTreeState:"clean", BuildDate:"2018-04-19T00:05:56Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.7", GitCommit:"dd5e1a2978fd0b97d9b78e1564398aeea7e7fe92", GitTreeState:"clean", BuildDate:"2018-04-18T23:58:35Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

Cloud Provider/Platform (AKS, GKE, Minikube etc.): AWS

[stevvooe]

I've been looking into this recently and it looks like there are a few major problems that prevent one from doing a secure tiller setup:

1. The base tiller configuration uses kubectl proxy. This will invalidate any tls setup available, making --tls-verify unusable without the --host flag.
2. To use the --host flag, one must expose tiller securely outside the cluster. This can be via ingress or some other proxy that supports http2/grpc. Because this is an RBAC-ignoring, permission elevating API, doing this correctly is a huge risk.
3. Setup of --tls isn't very useful without --tls-verify. If one cannot verify the remote resource, encrypting the data isn't very useful, as you can't really trust who you are talking to.

Given this, it might be good to update the documentation to reflect that configuring tiller to work with tls, one must implement step 2 above. I don't have a good suggestion for those steps.

The other option is to provide a solution for tls over kubectl proxy, which might be more work than it is worth.

Now, I understand that helm v3 will address many of the problems above, but it might be good to warn users of the security implications of running tiller in their cluster. It seems like most tiller installations are likely very insecure.

[fibonacci1729]

When generating your tiller certificate, you can try adding an IP SAN for 127.0.0.1.

For example:

$ echo subjectAltName=IP:127.0.0.1 > extfile.cnf
$ openssl x509 -req -CA ca.cert.pem -CAkey ca.key.pem -CAcreateserial -in tiller.csr.pem -out tiller.cert.pem -days 365 -extfile extfile.cnf

Because of the tunnel (kube proxy), the client sees the host as 127.0.0.1 during TLS hostname verification.

Updating the helm docs to reflect this.

[stevvooe]

@fibonacci1729 Unfortunately, the adding the IP san for localhost does not work.