Set etcd pod cipher suites per OpenShift 3.10 documentation

[bbeaudoin]

Setting the etcd cipher suites does not work as documented (the etcd.yaml does not specify the --cipher-suites option and does not read values from /etc/origin/master/master-config.yaml).

https://docs.openshift.com/container-platform/3.10/install_config/master_node_configuration.html#master-config-tls-cipher

This change converts files/etcd.yaml to a template in templates/etcd.yaml.j2 and appends the correct option to the command entrypoint.

Without this change, the issue described in etcd-io/etcd#8320 still exists as the cipher suite list remains autopopulated by go without a whitelist. This addresses the following bug: https://bugzilla.redhat.com/show_bug.cgi?id=1623646.

[openshift-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: bbeaudoin
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approver: vrutkovs

If they are not already assigned, you can assign the PR to them by writing /assign @vrutkovs in a comment when ready.

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-ci-robot]

@bbeaudoin: The following test failed, say /retest to rerun them all:

Test name	Commit	Details	Rerun command
ci/prow/gcp	e510f58	link	/test gcp

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[sdodson]

Wouldn't it be much easier to set environment variables in /etc/etcd/etcd.conf?

[DanyC97]

nit: you could squash the 2 commits into one @bbeaudoin

[vrutkovs]

Agree with @sdodson - mixing conf file and cli params is unmaintainable

[bbeaudoin]

@sdodson @vrutkovs unfortunately setting ETCD_CIPHER_SUITES using the environment variable or etcd.conf causes etcd-3.2.22 to crash without logs and the pod goes into a crash loop.

This method is the only method known to work at this time.

[vrutkovs]

unfortunately setting ETCD_CIPHER_SUITES using the environment variable or etcd.conf causes etcd-3.2.22 to crash without logs and the pod goes into a crash loop.

Weird, it worked fine here, see #9883, works with etcd_cipher_suites="TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256"

Did you use the image from quay.io or registry.access.redhat.com?

[sdodson]

I'll do a quick test in my OCP 3.10 cluster.

[bbeaudoin]

@vrutkovs @sdodson I've retested against the solution at
https://access.redhat.com/solutions/3499651 and found it to be working as documented.

Last week the pod was consistently exiting without explanation in the Docker logs and system logs reporting

Aug 28 18:48:09 master-310-1 atomic-openshift-node: I0828 18:48:09.479368 1874 kubelet.go:1923] SyncLoop (PLEG): "master-etcd-master-310-1.example.com_kube-system(e8223b6b4c7c540f9b44445601438d24)", event: &pleg.PodLifecycleEvent{ID:"e8223b6b4c7c540f9b44445601438d24", Type:"ContainerDied", Data:"7ececb5eddffba651220578c5ff7e121e8e9cdd73a42b929a8efc96d00a3351d"}

This week I am unable to reproduce the issue. Checked against #9883, that solution is working. Closing this PR and suggesting we move forward with the solution proposed by @vrutkovs.

[sdodson]

As long as the value of the envrionment variable is quoted it works, here I connect with DES-CBC3-SHA, add the envvar, restart, and show that it fails to handshake afterwards.

[root@ose3-master ~]# openssl s_client -cipher DES-CBC3-SHA -connect `hostname`:2379                                                                                                                               
CONNECTED(00000003)
depth=0 CN = ose3-master.example.com
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = ose3-master.example.com
verify error:num=21:unable to verify the first certificate
verify return:1
140258724403088:error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate:s3_pkt.c:1493:SSL alert number 42
140258724403088:error:140790E5:SSL routines:ssl23_write:ssl handshake failure:s23_lib.c:177:
---
Certificate chain
 0 s:/CN=ose3-master.example.com
   i:/CN=etcd-signer@1535390245
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIEmDCCAoCgAwIBAgIBBzANBgkqhkiG9w0BAQsFADAhMR8wHQYDVQQDDBZldGNk
LXNpZ25lckAxNTM1MzkwMjQ1MB4XDTE4MDgyNzE4MTAyN1oXDTIzMDgyNjE4MTAy
N1owIjEgMB4GA1UEAwwXb3NlMy1tYXN0ZXIuZXhhbXBsZS5jb20wggEiMA0GCSqG
SIb3DQEBAQUAA4IBDwAwggEKAoIBAQCrzKJMEm1AlQDVjyqUYrWtZTxjbzU3DitB
1TBV7SxKgm0Cu0xmyyf96APNFqkXQGHiI14zlNjaFBigXaA7+gVmV6u5o1SQZxkU
qOnC3XIghQ0R33HAhrnjCQuSQprIfV1kA3F4kIM73BmecOrYUKRxhAVDCYiXgGWB
Fah1vEZeq2U57FPHncay8AjzWruNFtruVf61GKZG3DRDuIRbhnsdfsmU//u/ig4r
48CqhjnM2IzyLoiRWuKAqMQWha23r/JN4+qOp0kR+LgxCWPXRqu+oK3GYMusZXyQ
QEn96V8KUu2r4OJI4+pPCfkEQZm23LMOTMWQt8uSQ8flXbAd/kh3AgMBAAGjgdkw
gdYwUQYDVR0jBEowSIAUTxM8Efx8w2ZSFiCAGoz9kcGQKHmhJaQjMCExHzAdBgNV
BAMMFmV0Y2Qtc2lnbmVyQDE1MzUzOTAyNDWCCQCPEw1MwKQAJzAMBgNVHRMBAf8E
AjAAMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjALBgNVHQ8EBAMCBaAw
HQYDVR0OBBYEFKgy/WIS9Z8J4GSKetDnWe/9u54yMCgGA1UdEQQhMB+HBMCoejSC
F29zZTMtbWFzdGVyLmV4YW1wbGUuY29tMA0GCSqGSIb3DQEBCwUAA4ICAQACvyEU
Kf3Fwm983a8Iu4hT8OryxYY4qKgZti86DGwaf1fByT2+3eqo2X4AzpvOrCPsaBJh
FDsHnJlMgGtf28ecnJem4bR7Bd9dJyBY7h7kTzEPMpB83Xq/GKrOarOlNEmtveZ0
Sw7mGFd5m6jaKqnxluIMenG1FzEeDSIRHdocn6SR0zmA0QmSnXRAYsl51d7KABvG
wMpS/l6br5C/i6lgvyPhJoGWU1oBk404Rmj+2OAZ0Vkp/LRsPy8Jimo1jPwEEzOT
McIGUDczltDCXp0n56SKpYg+74enQ6RxhPQsUh5zo6XYNj68urYDDiWQogOZS1Qb
DaiWi2X8uCfXrRtW525rRSsQMnOaEjhlqmDsiQlCbaxMBAYEmprB5kZOfz9xGHWy
eMdJnPeIrR3GeB136tUHdGG2r/JB6xmCDGQ7bX4F8CqanlbXDG4WmiFNdS+g2GWb
4pvo4/ZH9cTS27fdi0Lzyi1jjbjT0ToJh8m1Y9EhQVHkbZr2cZS9lQoA6WjWBJML
d7m4kyHFm3yPBMhM8yK2BafxRzEy6ccWVnD5LmEwbh28v5JPSwDt+a/cTD4IL+hH
PdIoDQSO2DH3KHVrktzU0i6KGtwPyuKkKfdqxj73QU2NTej6xopwYaG6N5Nr9WgC
+NWykTh6PsF9eu1qoq5SZlMPF+sW0T3YHDMwUw==
-----END CERTIFICATE-----
subject=/CN=ose3-master.example.com
issuer=/CN=etcd-signer@1535390245
---
Acceptable client certificate CA names
/CN=etcd-signer@1535390245
Client Certificate Types: RSA sign, ECDSA sign
Requested Signature Algorithms: RSA+SHA256:ECDSA+SHA256:RSA+SHA384:ECDSA+SHA384:RSA+SHA1:ECDSA+SHA1
Shared Requested Signature Algorithms: RSA+SHA256:ECDSA+SHA256:RSA+SHA384:ECDSA+SHA384:RSA+SHA1:ECDSA+SHA1
---
SSL handshake has read 1334 bytes and written 338 bytes
---
New, TLSv1/SSLv3, Cipher is DES-CBC3-SHA
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : DES-CBC3-SHA
    Session-ID: 
    Session-ID-ctx: 
    Master-Key: DF2CEA92E8275A578B00910B25B19F448399664528B5A5045CD9D53D2752129D6DEBDE8C12C9F82E2A5BF42405BA4256
    Key-Arg   : None
    Krb5 Principal: None
    PSK identity: None
    PSK identity hint: None
    Start Time: 1536070386
    Timeout   : 300 (sec)
    Verify return code: 21 (unable to verify the first certificate)

[root@ose3-master ~]# echo 'ETCD_CIPHER_SUITES="TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256"' >> /etc/etcd/etcd.conf

[root@ose3-master ~]# master-restart etcd
0

[root@ose3-master ~]# openssl s_client -cipher DES-CBC3-SHA -connect `hostname`:2379
CONNECTED(00000003)
139658369894288:error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure:s23_clnt.c:769:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 99 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : 0000
    Session-ID: 
    Session-ID-ctx: 
    Master-Key: 
    Key-Arg   : None
    Krb5 Principal: None
    PSK identity: None
    PSK identity hint: None
    Start Time: 1536070427
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---

[sdodson]

It definitely caused the pod to crashloop if the value was unquoted.

[bbeaudoin]

@sdodson thanks for figuring that out, a quoting issue wasn’t suspected at all. That will save some pain later as will the PR @vrutkovs added to change the etcd.conf instead.