Init websocket connection failed remote error: tls: handshake failure

[luogangyi]

What happened:
edgecore cannot connect to cloudcore. I have try both websocket and quic.
logs as below

May 19 07:38:32 edgenode1 edgecore[2055]: I0519 07:38:32.015127    2055 edged.go:329] starting plugin manager
May 19 07:38:32 edgenode1 edgecore[2055]: I0519 07:38:32.015136    2055 edged.go:332] starting syncPod
May 19 07:38:32 edgenode1 edgecore[2055]: I0519 07:38:32.015211    2055 plugin_manager.go:114] Starting Kubelet Plugin Manager
May 19 07:38:32 edgenode1 edgecore[2055]: I0519 07:38:32.015454    2055 record.go:19] Normal NodeAllocatableEnforced Updated Node Allocatable limit across pods
May 19 07:38:32 edgenode1 edgecore[2055]: I0519 07:38:32.015569    2055 volume_manager.go:265] Starting Kubelet Volume Manager
May 19 07:38:32 edgenode1 edgecore[2055]: I0519 07:38:32.017271    2055 server.go:35] starting to listen read-only on 127.0.0.1:10350
May 19 07:38:32 edgenode1 edgecore[2055]: I0519 07:38:32.017888    2055 server.go:354] Adding debug handlers to kubelet server.
May 19 07:38:32 edgenode1 edgecore[2055]: I0519 07:38:32.018282    2055 desired_state_of_world_populator.go:138] Desired state populator starts to run
May 19 07:38:32 edgenode1 edgecore[2055]: I0519 07:38:32.035586    2055 edged_status.go:392] Attempting to register node edgenode1
May 19 07:38:32 edgenode1 edgecore[2055]: I0519 07:38:32.218857    2055 reconciler.go:156] Reconciler: start to sync state
May 19 07:38:36 edgenode1 edgecore[2055]: E0519 07:38:36.895729    2055 ws.go:79] dial websocket error(remote error: tls: handshake failure), response message:
May 19 07:38:36 edgenode1 edgecore[2055]: E0519 07:38:36.895764    2055 websocket.go:90] Init websocket connection failed remote error: tls: handshake failure
May 19 07:38:41 edgenode1 edgecore[2055]: E0519 07:38:41.897666    2055 ws.go:79] dial websocket error(remote error: tls: handshake failure), response message:
May 19 07:38:41 edgenode1 edgecore[2055]: E0519 07:38:41.897722    2055 websocket.go:90] Init websocket connection failed remote error: tls: handshake failure
May 19 07:38:42 edgenode1 edgecore[2055]: I0519 07:38:42.015329    2055 process.go:675] get a message {Header:{ID:81da68cb-3bb8-49c5-bb81-5cc2e867f543 ParentID: Timestamp:1589888322015 ResourceVersion: Sync:false} Router:{Source:edged Group:edged Operation:query Resource:default/pod} Content:<nil>}
May 19 07:38:42 edgenode1 edgecore[2055]: I0519 07:38:42.015772    2055 edged.go:1008] result content is []
May 19 07:38:46 edgenode1 edgecore[2055]: E0519 07:38:46.899385    2055 ws.go:79] dial websocket error(remote error: tls: handshake failure), response message:
May 19 07:38:46 edgenode1 edgecore[2055]: E0519 07:38:46.899434    2055 websocket.go:90] Init websocket connection failed remote error: tls: handshake failure
May 19 07:38:51 edgenode1 edgecore[2055]: E0519 07:38:51.900953    2055 ws.go:79] dial websocket error(remote error: tls: handshake failure), response message:
May 19 07:38:51 edgenode1 edgecore[2055]: E0519 07:38:51.900972    2055 websocket.go:90] Init websocket connection failed remote error: tls: handshake failure

What you expected to happen:
edgecore connected to cloudcore.

How to reproduce it (as minimally and precisely as possible):
use latest version of kubeedge(version >= 1.3)
and use auto Certificates generating feature.

Anything else we need to know?:

Environment:

• KubeEdge version(e.g. cloudcore/edgecore --version):
  KubeEdge v1.3.0-4+a2f16443495233-dirty

CloudSide Environment:

• OS (e.g. cat /etc/os-release): CentOS 7
• OpenSSL: OpenSSL 1.0.2k-fips 26 Jan 2017

EdgeSide Environment:

• OS (e.g. cat /etc/os-release): CentOS 7
• OpenSSL: OpenSSL 1.0.2k-fips 26 Jan 2017

[luogangyi]

I have inspect the CA and certificates are correctly fetch by edgecore.
I try to use openssl manually,

[root@edgenode1 ~]# openssl s_client -servername masternode1 -connect 192.168.96.94:10000 -cert /etc/kubeedge/certs/server.crt -key /etc/kubeedge/certs/server.key -CAfile /etc/kubeedge/ca/rootCA.crt -state
CONNECTED(00000003)
SSL_connect:before/connect initialization
SSL_connect:SSLv2/v3 write client hello A
SSL3 alert read:fatal:handshake failure
SSL_connect:error in SSLv2/v3 read server hello A
140098377533328:error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure:s23_clnt.c:769:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 309 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : 0000
    Session-ID:
    Session-ID-ctx:
    Master-Key:
    Key-Arg   : None
    Krb5 Principal: None
    PSK identity: None
    PSK identity hint: None
    Start Time: 1589942761
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---

[luogangyi]

more digging,
Cloudhub use below method to config tls.
note TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 are configured as only cipher.

func createTLSConfig(ca, cert, key []byte) tls.Config {
	// init certificate
	pool := x509.NewCertPool()
	ok := pool.AppendCertsFromPEM(pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: ca}))
	if !ok {
		panic(fmt.Errorf("fail to load ca content"))
	}

	certificate, err := tls.X509KeyPair(pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: cert}), pem.EncodeToMemory(&pem.Block{Type: "PRIVATE KEY", Bytes: key}))
	if err != nil {
		panic(err)
	}
	return tls.Config{
		ClientCAs:    pool,
		ClientAuth:   tls.RequireAndVerifyClientCert,
		Certificates: []tls.Certificate{certificate},
		MinVersion:   tls.VersionTLS12,
		CipherSuites: []uint16{tls.TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256},
	}
}

However, ECDSA are use to generate key, see below.

// NewPrivateKey creates an RSA private key
func NewPrivateKey() (crypto.Signer, error) {
	return ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
}

ECDHA key(and certificates) is not match RSA cipher, see openssl notes

A RSA cipher can only be chosen, when a RSA certificate is available.
RSA export ciphers with a keylength of 512 bits for the RSA key require
a temporary 512 bit RSA key, as typically the supplied key has a length
of 1024 bit (see
L<SSL_CTX_set_tmp_rsa_callback(3)|SSL_CTX_set_tmp_rsa_callback(3)>).
RSA ciphers using DHE need a certificate and key and additional DH-parameters
(see L<SSL_CTX_set_tmp_dh_callback(3)|SSL_CTX_set_tmp_dh_callback(3)>).

A DSA cipher can only be chosen, when a DSA certificate is available.
DSA ciphers always use DH key exchange and therefore need DH-parameters
(see L<SSL_CTX_set_tmp_dh_callback(3)|SSL_CTX_set_tmp_dh_callback(3)>).

When these conditions are not met for any cipher in the list (e.g. a
client only supports export RSA ciphers with a asymmetric key length
of 512 bits and the server is not configured to use temporary RSA
keys), the "no shared cipher" (SSL_R_NO_SHARED_CIPHER) error is generated
and the handshake will fail.

So, I believe the mismatch of cipher and certificates caused tls handshake failure

[luogangyi]

/assign

[GsssC]

@XJangel @ls889

[yzhao66]

so,how to resolve this proble?

[ls889]

so,how to resolve this proble?

Pr1702