Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gRPC don't work with SSL/TLS #6951

Closed
Felixoid opened this issue Dec 6, 2016 · 8 comments

Comments

2 participants
@Felixoid
Copy link
Contributor

commented Dec 6, 2016

Version

etcd Version: 3.0.15
Git SHA: GitNotFound
Go Version: go1.7.3
Go OS/Arch: linux/amd64

First part: noising logs from v3rpc/grpc

Config:

name: host01
data-dir: /var/lib/etcd/test-ssl-auto
listen-peer-urls: https://host01.domain.yandex.net:2480
listen-client-urls: https://host01.domain.yandex.net:2479,https://localhost:2479
initial-advertise-peer-urls: https://host01.domain.yandex.net:2480
initial-cluster: host01=https://host01.domain.yandex.net:2480,host02=https://host02.domain.yandex.net:2480,host03=https://host03.domain.yandex.net:2480
initial-cluster-state: new
initial-cluster-token: test-ssl
advertise-client-urls: https://host01.domain.yandex.net:2479
client-transport-security:
  auto-tls: true
peer-transport-security:
  auto-tls: true
client-cert-auth: false

http2Client.notify.txt
After another server connect with enabled ssl begin to repeat string:
2016-12-06 16:31:43.093112 I | v3rpc/grpc: transport: http2Client.notifyError got notified that the client transport was broken unexpected EOF.
I think it's the root of the next problem

Second part: etcdctl didn't work with enabled encryption

ok, drop database, try again with | grep -v
Config:

name: host01
data-dir: /var/lib/etcd/test-ssl-auto
listen-peer-urls: https://host01.domain.yandex.net:2480
listen-client-urls: https://host01.domain.yandex.net:2479,https://localhost:2479
initial-advertise-peer-urls: https://host01.domain.yandex.net:2480
initial-cluster: host01=https://host01.domain.yandex.net:2480,host02=https://host02.domain.yandex.net:2480,host03=https://host03.domain.yandex.net:2480
initial-cluster-state: new
initial-cluster-token: test-ssl
advertise-client-urls: https://host01.domain.yandex.net:2479
client-transport-security:
  auto-tls: true
peer-transport-security:
  auto-tls: true
client-cert-auth: false

auto-tls.txt
auto-tls-etcdctl.txt

Reproduced also with valid cert+key+cacert
Config:

data-dir: /var/lib/etcd/test-ssl
listen-peer-urls: https://host01.domain.yandex.net:2480
listen-client-urls: https://host01.domain.yandex.net:2479,https://localhost:2479
initial-advertise-peer-urls: https://host01.domain.yandex.net:2480
initial-cluster: host01=https://host01.domain.yandex.net:2480,host02=https://host02.domain.yandex.net:2480,host03=https://host03.domain.yandex.net:2480
initial-cluster-state: new
initial-cluster-token: test-ssl
advertise-client-urls: https://host01.domain.yandex.net:2479
client-transport-security:
  cert-file: /tmp/test-ssl/certs/etcd.pem
  key-file: /tmp/test-ssl/certs/etcd.key
peer-transport-security:
  cert-file: /tmp/test-ssl/certs/etcd.pem
  key-file: /tmp/test-ssl/certs/etcd.key
  trusted-ca-file: /tmp/test-ssl/certs/caroot.crt
client-cert-auth: false

tls.txt
tls-etcdctl.txt

Trace of the server with the last config
trace.txt

@Felixoid Felixoid changed the title gRPC with SSL/TLS gRPC don't work with SSL/TLS Dec 6, 2016

@gyuho

This comment has been minimized.

Copy link
Member

commented Dec 6, 2016

Can you try specifying your key as well?

Something like

ETCDCTL_API=3 /etcdctl \
    --endpoints 10.240.0.35:2379,10.240.0.36:2379,10.240.0.37:2379 \
    --cacert /etc/ssl/certs/etcd-root-ca.pem \
    --cert /etc/ssl/certs/my-etcd-1.pem \
    --key /etc/ssl/certs/my-etcd-1-key.pem \
    endpoint health

I see only --cacert flag in your etcdctl command.

@Felixoid

This comment has been minimized.

Copy link
Contributor Author

commented Dec 6, 2016

I think that for traffic encryption we don't need use certificate authorization.
We tried to use case from this example https://coreos.com/etcd/docs/latest/security.html#example-1-client-to-server-transport-security-with-https

And --cacert is used for validate selfsigned server's certificate

@gyuho

This comment has been minimized.

Copy link
Member

commented Dec 6, 2016

Oh I see client-cert-auth: false now. You are right. It should work.
I just tested the same setup in localhost and it works for me.
I generated certs with cfssl signing my localhost.

How did you generate certs?
Can you double-check it's valid for your hosts?

Thanks.

@Felixoid

This comment has been minimized.

Copy link
Contributor Author

commented Dec 7, 2016

Good morning.
The certs was generated with the ssl.cnf
commandline:

$ openssl version 
OpenSSL 1.0.1f 6 Jan 2014
$ openssl req -x509 -new -nodes -newkey rsa:2048 -keyout rootCA.key -sha256 -days 1024 -out caroot.crt -config ssl.cnf
Generating a 2048 bit RSA private key
............+++
.....+++
writing new private key to 'rootCA.key'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [RU]:
State or Province Name (full name) [Russia]:
Locality Name (eg, city) [Moscow]:
Organization Name (eg, company) [yyy]:
Organizational Unit Name (eg, section) [testing]:
Common Name (eg, YOUR name) [etcd]:
Email Address [sdfghjk@lkjhgfd.as]:
$ openssl req -extensions v3_req -new -newkey rsa:2048 -keyout etcd.key  -out etcd.csr -sha256 -nodes -config ssl.cnf
Generating a 2048 bit RSA private key
.........................+++
..........+++
writing new private key to 'etcd.key'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [RU]:
State or Province Name (full name) [Russia]:
Locality Name (eg, city) [Moscow]:
Organization Name (eg, company) [yyy]:
Organizational Unit Name (eg, section) [testing]:
Common Name (eg, YOUR name) [etcd]:
Email Address [sdfghjk@lkjhgfd.as]:
$ openssl x509 -req -in etcd.csr -CA caroot.crt -CAkey rootCA.key -CAcreateserial -out etcd.pem -days 5000 -extfile ssl.cnf -extensions v3_req
Signature ok
subject=/C=RU/ST=Russia/L=Moscow/O=yyy/OU=testing/CN=etcd/emailAddress=sdfghjk@lkjhgfd.as
Getting CA Private Key
$ openssl x509 -in /tmp/test-ssl/certs/etcd.pem -noout -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 12976209017358108321 (0xb414c159b2d9c6a1)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=RU, ST=Russia, L=Moscow, O=yyy, OU=testing, CN=etcd/emailAddress=sdfghjk@lkjhgfd.as
        Validity
            Not Before: Dec  6 22:38:12 2016 GMT
            Not After : Aug 15 22:38:12 2030 GMT
        Subject: C=RU, ST=Russia, L=Moscow, O=yyy, OU=testing, CN=etcd/emailAddress=sdfghjk@lkjhgfd.as
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:e1:5e:7b:3c:37:1d:64:73:64:91:3f:c3:a1:7b:
                    ae:9c:dc:2e:f8:1f:53:cc:b0:04:22:f0:f7:56:5d:
                    01:5d:12:d7:5d:a8:f9:33:e3:29:e9:6a:1e:b1:1f:
                    9c:88:a2:84:8b:7e:08:e4:f8:e7:de:6c:ef:ac:61:
                    52:e6:ec:2d:c5:d2:07:1d:55:92:df:e4:d2:b0:fd:
                    ac:7d:c8:da:40:8a:cc:97:50:25:06:bb:30:e8:49:
                    d0:dd:62:fa:36:ec:1b:e4:cf:88:b6:ec:ee:49:63:
                    de:a9:90:24:a2:a4:26:0e:8e:d7:e5:c7:da:d5:ad:
                    f4:d6:90:ce:6c:80:b2:25:35:d3:23:93:21:b6:22:
                    c2:f3:31:98:8c:8e:3c:0e:d8:c9:be:c7:ca:45:d2:
                    d6:a3:8a:56:95:52:37:f0:c5:ff:dd:7d:11:3c:b7:
                    93:7e:5d:a4:47:75:e6:29:d9:3c:f5:e1:a2:ad:df:
                    1f:97:79:73:91:48:6e:f4:b8:83:54:03:c3:8c:83:
                    9a:37:a1:04:f4:5d:c0:6d:f7:a9:a3:69:5a:99:a6:
                    b6:fd:bd:9b:8c:c3:2c:8f:db:f3:64:9f:c5:60:6b:
                    54:88:f0:b4:6d:91:5f:da:f5:a5:6e:37:ae:37:ce:
                    6a:63:e0:4b:23:44:56:0c:bc:7d:aa:31:53:43:bd:
                    ef:65
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Basic Constraints: 
                CA:FALSE
            X509v3 Key Usage: 
                Digital Signature, Non Repudiation, Key Encipherment
            X509v3 Subject Alternative Name: 
                DNS:host01.domain.yandex.net, DNS:host02.domain.yandex.net, DNS:host03.domain.yandex.net, DNS:localhost
    Signature Algorithm: sha256WithRSAEncryption
         6a:b3:dc:c1:b9:01:06:31:f0:52:38:be:bf:c6:12:fa:42:2c:
         f4:c1:fa:f7:e0:61:b5:e5:f3:f7:86:22:42:75:b6:03:29:32:
         a5:26:6f:ed:7b:50:16:64:57:d2:15:2d:a8:af:d5:1c:b2:e3:
         08:59:a7:95:af:94:c0:ad:56:ee:c7:6f:34:bb:54:43:e1:86:
         54:96:1e:92:ed:a3:d8:da:b3:86:d1:b2:d4:a4:f7:8d:5b:5b:
         e4:fe:f7:ac:3d:34:c9:e4:ba:23:ef:cd:0e:c1:04:77:4c:fb:
         65:99:e3:8c:aa:6a:50:f8:f9:37:7b:7b:f6:cc:8b:38:98:a6:
         42:d8:1e:40:6b:3b:c8:c2:f2:e7:fd:ad:f2:d2:b4:51:10:ec:
         f0:62:79:93:6b:fc:88:d9:47:92:fb:1d:0c:77:e5:a3:c0:25:
         19:53:ed:22:7f:b2:d9:bb:b5:c1:a9:96:32:fb:48:8f:bf:ff:
         ff:2e:3b:7c:d0:9e:78:d8:c3:2e:e6:ac:e4:dd:f1:4d:b5:ff:
         6c:c5:8a:65:46:a5:a9:a3:a9:e7:83:8f:c4:f8:32:e5:bf:f6:
         7b:55:4b:56:62:00:18:3b:22:17:ba:f6:b4:94:b8:db:fe:e5:
         f5:2d:02:d5:d1:8e:e6:5f:64:7a:89:05:95:8f:c6:19:b6:22:
         ce:32:3c:00

Then these files was synced on all hosts.

Then on host02 I launch etcd with --debug flag, logs
If certcs were invalid, then the debug logs would have been different, some like
2016-12-07 13:27:32.385448 D | rafthttp: failed to dial 5f8e785a7ec1d5e0 on stream MsgApp v2 (x509: certificate is valid for host01.another-domain.yandex.net, host02.another-domain.yandex.net, host03.another-domain.yandex.net, etcd-test.another-domain.yandex.net, not host03.domain.yandex.net)
In the end of logs there attempts to connect etcdctl from host01

$ export ETCDCTL_API=3
$ etcdctl --endpoints=https://host01.domain.yandex.net:2479,https://host02.domain.yandex.net:2479,https://host03.domain.yandex.net:2479 --cacert /tmp/test-ssl/certs/caroot.crt member list
Error:  context deadline exceeded
@gyuho

This comment has been minimized.

Copy link
Member

commented Dec 7, 2016

@Felixoid Can you try our master branch or v3.1.x?
https://github.com/coreos/etcd/releases/tag/v3.1.0-rc.1

Could be Go related...
This issue sounds similar to this one #6565 (comment).

And I notice

etcd Version: 3.0.15
Git SHA: GitNotFound
Go Version: go1.7.3
Go OS/Arch: linux/amd64

This doesn't seem like our official release. We build v3.0.x with Go 1.6.3.

Can you also try our official etcd releases?
Thanks.

@gyuho

This comment has been minimized.

Copy link
Member

commented Dec 9, 2016

@Felixoid Any updates?

@Felixoid

This comment has been minimized.

Copy link
Contributor Author

commented Dec 9, 2016

Hello @gyuho
Sorry for a long silence!
thanks, official release etcd-v3.0.15-linux-amd64.tar.gz works properly!

etcd Version: 3.0.15
Git SHA: fc00305
Go Version: go1.6.3
Go OS/Arch: linux/amd64
etcdctl --endpoints=https://host01.domain.yandex.net:2479,https://host02.domain.yandex.net:2479,https://host03.domain.yandex.net:2479 --cacert /tmp/test-ssl/certs/caroot.crt member list
1e304c4f8c8579c0, started, host01, https://host01.domain.yandex.net:2480, https://host01.domain.yandex.net:2479
4dbbf57aeab16382, started, host02, https://host02.domain.yandex.net:2480, https://host02.domain.yandex.net:2479
5f8e785a7ec1d5e0, started, host03, https://host03.domain.yandex.net:2480, https://host03.domain.yandex.net:2479

I built different versions of releases etcd + golang for experiments:

/tmp/etcd-3.0.15_1.6.4/etcd --version
etcd Version: 3.0.15
Git SHA: GitNotFound
Go Version: go1.6.4
Go OS/Arch: linux/amd64

All works fine

/tmp/etcd-v3.1.0-rc.1/bin/etcd --version
etcd Version: 3.1.0-rc.1
Git SHA: GitNotFound
Go Version: go1.7.4
Go OS/Arch: linux/amd64

Also works as expected.

I think it's solution - don't to build v3.0 with go v1.7

@gyuho

This comment has been minimized.

Copy link
Member

commented Dec 9, 2016

Yeah v3.0.x should be used with Go 1.6.x.
And v3.1.x will be released with Go 1.7+

Closing.

Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.