etcd curl: (58) unable to load client key: -8178 (SEC_ERROR_BAD_KEY)

[froblestecval]

Environmental Info:
K3s Version:

k3s version v1.29.4+k3s1 (94e29e2)
go version go1.21.9

Node(s) CPU architecture, OS, and Version:

Cluster Configuration:

3 masters
3 workers

Describe the bug:

I can't obtain metrics etcd. Is ocurr after update from:
k3s version v1.25.10+k3s1 (613a3bc)
go version go1.19.9

Steps To Reproduce:

# curl https://localhost:2379/metrics --cacert /var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --cert /var/lib/rancher/k3s/server/tls/etcd/client.crt --key /var/lib/rancher/k3s/server/tls/etcd/client.key

• Installed K3s:

v1.29.4+k3s1

Expected behavior:

Output metrics: example:

# TYPE volume_manager_total_volumes gauge
volume_manager_total_volumes{plugin_name="kubernetes.io/host-path",state="actual_state_of_world"} 4
volume_manager_total_volumes{plugin_name="kubernetes.io/host-path",state="desired_state_of_world"} 4
volume_manager_total_volumes{plugin_name="kubernetes.io/secret",state="actual_state_of_world"} 1
volume_manager_total_volumes{plugin_name="kubernetes.io/secret",state="desired_state_of_world"} 1
.
.
ommit output

Actual behavior:

curl: (58) unable to load client key: -8178 (SEC_ERROR_BAD_KEY)

Additional context / logs:

[brandond]

RKE2 has always used EC keys. It sounds like the version of curl (or the version of openssl your curl is linked against) doesn't include support for EC keys. The request works for me - although I will note that the metrics are not served on the port you're trying:

root@k3s-server-1:/# curl -vs https://localhost:2379/metrics --cacert /var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --cert /var/lib/rancher/k3s/server/tls/etcd/client.crt --key /var/lib/rancher/k3s/server/tls/etcd/client.key
*   Trying 127.0.0.1:2379...
* Connected to localhost (127.0.0.1) port 2379 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /var/lib/rancher/k3s/server/tls/etcd/server-ca.crt
*  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, CERT verify (15):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=etcd-server
*  start date: May 13 16:54:52 2024 GMT
*  expire date: May 13 16:54:52 2025 GMT
*  subjectAltName: host "localhost" matched cert's "localhost"
*  issuer: CN=etcd-server-ca@1715619292
*  SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* Using Stream ID: 1 (easy handle 0x615150141eb0)
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> GET /metrics HTTP/2
> Host: localhost:2379
> user-agent: curl/7.81.0
> accept: */*
>
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Connection state changed (MAX_CONCURRENT_STREAMS == 4294967295)!
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
< HTTP/2 415
< content-type: application/grpc
< grpc-status: 3
< grpc-message: invalid gRPC request content-type ""
<
* Connection #0 to host localhost left intact

root@k3s-server-1:/# curl -s https://localhost:2382/metrics --cacert /var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --cert /var/lib/rancher/k3s/server/tls/etcd/client.crt --key /var/lib/rancher/k3s/server/tls/etcd/client.key | grep etcd_server_version
# HELP etcd_server_version Which version is running. 1 for 'server_version' label with current version.
# TYPE etcd_server_version gauge
etcd_server_version{server_version="3.5.9"} 1

[froblestecval]

What about prometheus? I must update de version so?

regards,

[brandond]

I don't know. Perhaps you need to change the scrape port in your prometheus config?

[froblestecval]

# curl https://localhost:2379/metrics --cacert /var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --cert /var/lib/rancher/k3s/server/tls/etcd/client.crt --key /var/lib/rancher/k3s/server/tls/etcd/client.key
curl: (58) unable to load client key: -8178 (SEC_ERROR_BAD_KEY)

[root@master01 ~]# curl --version
curl 7.29.0 (x86_64-redhat-linux-gnu) libcurl/7.29.0 NSS/3.44 zlib/1.2.7 libidn/1.28 libssh2/1.8.0
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smtp smtps telnet tftp 
Features: AsynchDNS GSS-Negotiate IDN IPv6 Largefile NTLM NTLM_WB SSL libz unix-sockets 

[brandond]

I can't help you if your version of curl or openssl doesn't support ec keys. You'll have to upgrade, or upgrade to a newer distro release that has a modern version of curl and openssl. I don't understand what this has to do with scraping things via Prometheus, either... prometheus doesn't use curl.

root@k3s-server-1:/# curl --version
curl 7.81.0 (x86_64-pc-linux-gnu) libcurl/7.81.0 OpenSSL/3.0.2 zlib/1.2.11 brotli/1.0.9 zstd/1.4.8 libidn2/2.3.2 libpsl/0.21.0 (+libidn2/2.3.2) libssh/0.9.6/openssl/zlib nghttp2/1.43.0 librtmp/2.3 OpenLDAP/2.5.17
Release-Date: 2022-01-05
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM NTLM_WB PSL SPNEGO SSL TLS-SRP UnixSockets zstd

root@k3s-server-1:/# cat /var/lib/rancher/k3s/server/tls/etcd/client.key
-----BEGIN EC PRIVATE KEY-----
MHcCAQEEIPEfgcXhWfJrnmjZ2KrJUQ4LKXUJJB/9m5reD34LLL4PoAoGCCqGSM49
AwEHoUQDQgAEQ8+TLFmZngs7rEIcj09V3uWwRbpVTR6rMz89HAmXPOS+cC8yLaZn
CALFl1Z4I38bTvrIpY50l+JLDJJ8cd/mRg==
-----END EC PRIVATE KEY-----

[froblestecval]

Thanks Brandon!

Very Good!