gRPC don't work with SSL/TLS

[Felixoid]

Version

etcd Version: 3.0.15
Git SHA: GitNotFound
Go Version: go1.7.3
Go OS/Arch: linux/amd64

First part: noising logs from v3rpc/grpc

Config:

name: host01
data-dir: /var/lib/etcd/test-ssl-auto
listen-peer-urls: https://host01.domain.yandex.net:2480
listen-client-urls: https://host01.domain.yandex.net:2479,https://localhost:2479
initial-advertise-peer-urls: https://host01.domain.yandex.net:2480
initial-cluster: host01=https://host01.domain.yandex.net:2480,host02=https://host02.domain.yandex.net:2480,host03=https://host03.domain.yandex.net:2480
initial-cluster-state: new
initial-cluster-token: test-ssl
advertise-client-urls: https://host01.domain.yandex.net:2479
client-transport-security:
  auto-tls: true
peer-transport-security:
  auto-tls: true
client-cert-auth: false

http2Client.notify.txt
After another server connect with enabled ssl begin to repeat string:
2016-12-06 16:31:43.093112 I | v3rpc/grpc: transport: http2Client.notifyError got notified that the client transport was broken unexpected EOF.
I think it's the root of the next problem

Second part: etcdctl didn't work with enabled encryption

ok, drop database, try again with | grep -v
Config:

name: host01
data-dir: /var/lib/etcd/test-ssl-auto
listen-peer-urls: https://host01.domain.yandex.net:2480
listen-client-urls: https://host01.domain.yandex.net:2479,https://localhost:2479
initial-advertise-peer-urls: https://host01.domain.yandex.net:2480
initial-cluster: host01=https://host01.domain.yandex.net:2480,host02=https://host02.domain.yandex.net:2480,host03=https://host03.domain.yandex.net:2480
initial-cluster-state: new
initial-cluster-token: test-ssl
advertise-client-urls: https://host01.domain.yandex.net:2479
client-transport-security:
  auto-tls: true
peer-transport-security:
  auto-tls: true
client-cert-auth: false

auto-tls.txt
auto-tls-etcdctl.txt

Reproduced also with valid cert+key+cacert
Config:

data-dir: /var/lib/etcd/test-ssl
listen-peer-urls: https://host01.domain.yandex.net:2480
listen-client-urls: https://host01.domain.yandex.net:2479,https://localhost:2479
initial-advertise-peer-urls: https://host01.domain.yandex.net:2480
initial-cluster: host01=https://host01.domain.yandex.net:2480,host02=https://host02.domain.yandex.net:2480,host03=https://host03.domain.yandex.net:2480
initial-cluster-state: new
initial-cluster-token: test-ssl
advertise-client-urls: https://host01.domain.yandex.net:2479
client-transport-security:
  cert-file: /tmp/test-ssl/certs/etcd.pem
  key-file: /tmp/test-ssl/certs/etcd.key
peer-transport-security:
  cert-file: /tmp/test-ssl/certs/etcd.pem
  key-file: /tmp/test-ssl/certs/etcd.key
  trusted-ca-file: /tmp/test-ssl/certs/caroot.crt
client-cert-auth: false

tls.txt
tls-etcdctl.txt

Trace of the server with the last config
trace.txt

[gyuho]

Can you try specifying your key as well?

Something like

ETCDCTL_API=3 /etcdctl \
    --endpoints 10.240.0.35:2379,10.240.0.36:2379,10.240.0.37:2379 \
    --cacert /etc/ssl/certs/etcd-root-ca.pem \
    --cert /etc/ssl/certs/my-etcd-1.pem \
    --key /etc/ssl/certs/my-etcd-1-key.pem \
    endpoint health

I see only --cacert flag in your etcdctl command.

[Felixoid]

I think that for traffic encryption we don't need use certificate authorization.
We tried to use case from this example https://coreos.com/etcd/docs/latest/security.html#example-1-client-to-server-transport-security-with-https

And --cacert is used for validate selfsigned server's certificate

[gyuho]

Oh I see client-cert-auth: false now. You are right. It should work.
I just tested the same setup in localhost and it works for me.
I generated certs with cfssl signing my localhost.

How did you generate certs?
Can you double-check it's valid for your hosts?

Thanks.

[Felixoid]

Good morning.
The certs was generated with the ssl.cnf
commandline:

$ openssl version 
OpenSSL 1.0.1f 6 Jan 2014
$ openssl req -x509 -new -nodes -newkey rsa:2048 -keyout rootCA.key -sha256 -days 1024 -out caroot.crt -config ssl.cnf
Generating a 2048 bit RSA private key
............+++
.....+++
writing new private key to 'rootCA.key'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [RU]:
State or Province Name (full name) [Russia]:
Locality Name (eg, city) [Moscow]:
Organization Name (eg, company) [yyy]:
Organizational Unit Name (eg, section) [testing]:
Common Name (eg, YOUR name) [etcd]:
Email Address [sdfghjk@lkjhgfd.as]:
$ openssl req -extensions v3_req -new -newkey rsa:2048 -keyout etcd.key  -out etcd.csr -sha256 -nodes -config ssl.cnf
Generating a 2048 bit RSA private key
.........................+++
..........+++
writing new private key to 'etcd.key'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [RU]:
State or Province Name (full name) [Russia]:
Locality Name (eg, city) [Moscow]:
Organization Name (eg, company) [yyy]:
Organizational Unit Name (eg, section) [testing]:
Common Name (eg, YOUR name) [etcd]:
Email Address [sdfghjk@lkjhgfd.as]:
$ openssl x509 -req -in etcd.csr -CA caroot.crt -CAkey rootCA.key -CAcreateserial -out etcd.pem -days 5000 -extfile ssl.cnf -extensions v3_req
Signature ok
subject=/C=RU/ST=Russia/L=Moscow/O=yyy/OU=testing/CN=etcd/emailAddress=sdfghjk@lkjhgfd.as
Getting CA Private Key
$ openssl x509 -in /tmp/test-ssl/certs/etcd.pem -noout -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 12976209017358108321 (0xb414c159b2d9c6a1)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=RU, ST=Russia, L=Moscow, O=yyy, OU=testing, CN=etcd/emailAddress=sdfghjk@lkjhgfd.as
        Validity
            Not Before: Dec  6 22:38:12 2016 GMT
            Not After : Aug 15 22:38:12 2030 GMT
        Subject: C=RU, ST=Russia, L=Moscow, O=yyy, OU=testing, CN=etcd/emailAddress=sdfghjk@lkjhgfd.as
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:e1:5e:7b:3c:37:1d:64:73:64:91:3f:c3:a1:7b:
                    ae:9c:dc:2e:f8:1f:53:cc:b0:04:22:f0:f7:56:5d:
                    01:5d:12:d7:5d:a8:f9:33:e3:29:e9:6a:1e:b1:1f:
                    9c:88:a2:84:8b:7e:08:e4:f8:e7:de:6c:ef:ac:61:
                    52:e6:ec:2d:c5:d2:07:1d:55:92:df:e4:d2:b0:fd:
                    ac:7d:c8:da:40:8a:cc:97:50:25:06:bb:30:e8:49:
                    d0:dd:62:fa:36:ec:1b:e4:cf:88:b6:ec:ee:49:63:
                    de:a9:90:24:a2:a4:26:0e:8e:d7:e5:c7:da:d5:ad:
                    f4:d6:90:ce:6c:80:b2:25:35:d3:23:93:21:b6:22:
                    c2:f3:31:98:8c:8e:3c:0e:d8:c9:be:c7:ca:45:d2:
                    d6:a3:8a:56:95:52:37:f0:c5:ff:dd:7d:11:3c:b7:
                    93:7e:5d:a4:47:75:e6:29:d9:3c:f5:e1:a2:ad:df:
                    1f:97:79:73:91:48:6e:f4:b8:83:54:03:c3:8c:83:
                    9a:37:a1:04:f4:5d:c0:6d:f7:a9:a3:69:5a:99:a6:
                    b6:fd:bd:9b:8c:c3:2c:8f:db:f3:64:9f:c5:60:6b:
                    54:88:f0:b4:6d:91:5f:da:f5:a5:6e:37:ae:37:ce:
                    6a:63:e0:4b:23:44:56:0c:bc:7d:aa:31:53:43:bd:
                    ef:65
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Basic Constraints: 
                CA:FALSE
            X509v3 Key Usage: 
                Digital Signature, Non Repudiation, Key Encipherment
            X509v3 Subject Alternative Name: 
                DNS:host01.domain.yandex.net, DNS:host02.domain.yandex.net, DNS:host03.domain.yandex.net, DNS:localhost
    Signature Algorithm: sha256WithRSAEncryption
         6a:b3:dc:c1:b9:01:06:31:f0:52:38:be:bf:c6:12:fa:42:2c:
         f4:c1:fa:f7:e0:61:b5:e5:f3:f7:86:22:42:75:b6:03:29:32:
         a5:26:6f:ed:7b:50:16:64:57:d2:15:2d:a8:af:d5:1c:b2:e3:
         08:59:a7:95:af:94:c0:ad:56:ee:c7:6f:34:bb:54:43:e1:86:
         54:96:1e:92:ed:a3:d8:da:b3:86:d1:b2:d4:a4:f7:8d:5b:5b:
         e4:fe:f7:ac:3d:34:c9:e4:ba:23:ef:cd:0e:c1:04:77:4c:fb:
         65:99:e3:8c:aa:6a:50:f8:f9:37:7b:7b:f6:cc:8b:38:98:a6:
         42:d8:1e:40:6b:3b:c8:c2:f2:e7:fd:ad:f2:d2:b4:51:10:ec:
         f0:62:79:93:6b:fc:88:d9:47:92:fb:1d:0c:77:e5:a3:c0:25:
         19:53:ed:22:7f:b2:d9:bb:b5:c1:a9:96:32:fb:48:8f:bf:ff:
         ff:2e:3b:7c:d0:9e:78:d8:c3:2e:e6:ac:e4:dd:f1:4d:b5:ff:
         6c:c5:8a:65:46:a5:a9:a3:a9:e7:83:8f:c4:f8:32:e5:bf:f6:
         7b:55:4b:56:62:00:18:3b:22:17:ba:f6:b4:94:b8:db:fe:e5:
         f5:2d:02:d5:d1:8e:e6:5f:64:7a:89:05:95:8f:c6:19:b6:22:
         ce:32:3c:00

Then these files was synced on all hosts.

Then on host02 I launch etcd with --debug flag, logs
If certcs were invalid, then the debug logs would have been different, some like
2016-12-07 13:27:32.385448 D | rafthttp: failed to dial 5f8e785a7ec1d5e0 on stream MsgApp v2 (x509: certificate is valid for host01.another-domain.yandex.net, host02.another-domain.yandex.net, host03.another-domain.yandex.net, etcd-test.another-domain.yandex.net, not host03.domain.yandex.net)
In the end of logs there attempts to connect etcdctl from host01

$ export ETCDCTL_API=3
$ etcdctl --endpoints=https://host01.domain.yandex.net:2479,https://host02.domain.yandex.net:2479,https://host03.domain.yandex.net:2479 --cacert /tmp/test-ssl/certs/caroot.crt member list
Error:  context deadline exceeded

[gyuho]

@Felixoid Can you try our master branch or v3.1.x?
https://github.com/coreos/etcd/releases/tag/v3.1.0-rc.1

Could be Go related...
This issue sounds similar to this one #6565 (comment).

And I notice

etcd Version: 3.0.15
Git SHA: GitNotFound
Go Version: go1.7.3
Go OS/Arch: linux/amd64

This doesn't seem like our official release. We build v3.0.x with Go 1.6.3.

Can you also try our official etcd releases?
Thanks.

[gyuho]

@Felixoid Any updates?

[Felixoid]

Hello @gyuho
Sorry for a long silence!
thanks, official release etcd-v3.0.15-linux-amd64.tar.gz works properly!

etcd Version: 3.0.15
Git SHA: fc00305
Go Version: go1.6.3
Go OS/Arch: linux/amd64

etcdctl --endpoints=https://host01.domain.yandex.net:2479,https://host02.domain.yandex.net:2479,https://host03.domain.yandex.net:2479 --cacert /tmp/test-ssl/certs/caroot.crt member list
1e304c4f8c8579c0, started, host01, https://host01.domain.yandex.net:2480, https://host01.domain.yandex.net:2479
4dbbf57aeab16382, started, host02, https://host02.domain.yandex.net:2480, https://host02.domain.yandex.net:2479
5f8e785a7ec1d5e0, started, host03, https://host03.domain.yandex.net:2480, https://host03.domain.yandex.net:2479

I built different versions of releases etcd + golang for experiments:

/tmp/etcd-3.0.15_1.6.4/etcd --version
etcd Version: 3.0.15
Git SHA: GitNotFound
Go Version: go1.6.4
Go OS/Arch: linux/amd64

All works fine

/tmp/etcd-v3.1.0-rc.1/bin/etcd --version
etcd Version: 3.1.0-rc.1
Git SHA: GitNotFound
Go Version: go1.7.4
Go OS/Arch: linux/amd64

Also works as expected.

I think it's solution - don't to build v3.0 with go v1.7

[gyuho]

Yeah v3.0.x should be used with Go 1.6.x.
And v3.1.x will be released with Go 1.7+

Closing.

Thanks!