SSL/TLS Connection Issue: SSL_ERROR_RX_RECORD_TOO_LONG with Self-Signed Certificate

[noor47-pixel]

I’m experiencing an issue with configuring TLS for the OpenStack exporter. Here’s a detailed description of the steps I’ve taken and the problem I’m encountering:

Steps Taken:

1. Generated a Self-Signed Certificate:

   • I created a self-signed certificate and private key using OpenSSL.

2. Updated Configuration File (web-config-file.yml):

   • I added the generated .crt and .key files to the tls_server_config section of the web-config-file.yml.

   tls_server_config:
     cert_file: /path/to/certificate.pem
     key_file: /path/to/private_key.pem
     #client_auth_type: NoClientCert
    

3. Configured the Prometheus Exporter:

   • The exporter is set to listen on port 8444 (since port 443 is already in use).

Issue:

When trying to access the exporter at https://ip:8444, I encounter the following error:

SSL_ERROR_RX_RECORD_TOO_LONG

The page cannot be displayed because the authenticity of the received data cannot be verified.

Logs:

The logs indicate that TLS is disabled:

ts=2024-07-30T10:27:29.545Z caller=main.go:64 level=info msg="Build context" build_context="(go=go1.18.10, platform=linux/amd64, user=, date=, tags=unknown)"
ts=2024-07-30T10:27:29.546Z caller=main.go:85 level=info msg="openstack exporter started in legacy mode"
ts=2024-07-30T10:27:29.547Z caller=tls_config.go:274 level=info msg="Listening on" address=[::]:9180
ts=2024-07-30T10:27:29.547Z caller=tls_config.go:277 level=info msg="TLS is disabled." http2=false address=[::]:9180

Additional Information:

• Port Configuration: I am using port 8444, not 443.
• TLS Configuration: Despite specifying certificates in the configuration, TLS appears to be disabled based on the log output.

Questions:

1. Why is TLS disabled even though the configuration file specifies certificates and the client_auth_type?
2. Could the issue be related to the certificate format or configuration? not shure but i don't know
3. What steps should I take to resolve the SSL_ERROR_RX_RECORD_TOO_LONG error?
4. Tls is disabled in log come from cloud.yamls where i have verify to false ?

Expected Behavior:

The exporter should correctly initiate TLS on port 8444 using the provided self-signed certificate and private key.

Attachments:

• Configuration file snippet
• Logs showing the TLS status

Thank you for your assistance.

[noor47-pixel]

@Jonher937 any idea, i also read that tls config is on beta version maybe it's not actually supported by the exporter?

[Jonher937]

Works fine on my end on latest master.

Steps to reproduce:

Generate cert

➜  openstack-exporter git:(main) ✗ openssl genrsa -out priv.key 2048
➜  openstack-exporter git:(main) ✗ openssl req -new -x509 -key priv.key -out cert.pem -days 360
< press enter on each for testing >
Country Name (2 letter code) [AU]:
State or Province Name (full name) [Some-State]:
Locality Name (eg, city) []:
Organization Name (eg, company) [Internet Widgits Pty Ltd]:
Organizational Unit Name (eg, section) []:
Common Name (e.g. server FQDN or YOUR name) []:
Email Address []:

Create exporter config and start exporter

➜  openstack-exporter git:(main) ✗ cat exporter.yaml 
tls_server_config:
  cert_file: cert.pem
  key_file: priv.key

➜  openstack-exporter git:(main) ✗ go run . --web.config.file=exporter.yaml                  
ts=2024-07-30T22:52:30.591Z caller=main.go:67 level=error msg="openstack-exporter: error: required argument 'cloud' or flag --multi-cloud not provided, try --help"
ts=2024-07-30T22:52:30.591Z caller=main.go:70 level=info msg="Build context" build_context="(go=go1.22.4, platform=linux/amd64, user=, date=, tags=unknown)"
ts=2024-07-30T22:52:30.591Z caller=main.go:154 level=info msg="openstack exporter started in legacy mode"
ts=2024-07-30T22:52:30.591Z caller=tls_config.go:274 level=info msg="Listening on" address=[::]:9180
ts=2024-07-30T22:52:30.591Z caller=tls_config.go:310 level=info msg="TLS is enabled." http2=true address=[::]:9180

Notice that the last line above says TLS is enabled.

curl the exporter

➜  openstack-exporter git:(main) ✗ curl -vk https://127.0.0.1:9180
*   Trying 127.0.0.1:9180...
* Connected to 127.0.0.1 (127.0.0.1) port 9180
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256 / x25519 / RSASSA-PSS
* ALPN: server accepted h2
* Server certificate:
*  subject: C=AU; ST=Some-State; O=Internet Widgits Pty Ltd
*  start date: Jul 30 22:52:20 2024 GMT
*  expire date: Jul 25 22:52:20 2025 GMT
*  issuer: C=AU; ST=Some-State; O=Internet Widgits Pty Ltd
*  SSL certificate verify result: self-signed certificate (18), continuing anyway.
*   Certificate level 0: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://127.0.0.1:9180/
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: 127.0.0.1:9180]
* [HTTP/2] [1] [:path: /]
* [HTTP/2] [1] [user-agent: curl/8.7.1]
* [HTTP/2] [1] [accept: */*]
> GET / HTTP/2
> Host: 127.0.0.1:9180
> User-Agent: curl/8.7.1
> Accept: */*
> 
* Request completely sent off
< HTTP/2 200 
< content-type: text/html; charset=UTF-8
< content-length: 938
< date: Tue, 30 Jul 2024 22:54:14 GMT
< 
<html>
....
.... removed for paste on Github
....
</html>
* Connection #0 to host 127.0.0.1 left intact

[noor47-pixel]

yes, but it doesn't accept my certificates. could this also be linked to the version used in the doc? I could see that we had TLS version minum at 1.2 and max 1.3.

@Jonher937 I've generated several certificates, and my certificate authority has also provided me with one, but it keeps telling me that the format of the certifiact is too long. Is this because I have the cert file in .crt instead of .pem like the link, or is it due to what? Thank you in advance for your reactivity on the subject.

[Jonher937]

How does your startup line for the exporter look? My guess is you are missing --web.config.file=web-config-file.yml since the exporter is reporting TLS is disabled.

[noor47-pixel]

@Jonher937 here is my create command from container inspect
"CreateCommand": [
"podman",
"container",
"run",
"--name",
"prometheus-openstack-exporter",
"--authfile",
"/etc/openstack/auth.json",
"--restart=always",
"--detach=True",
"--network",
"monitoring",
"--publish",
"8444:9180/tcp",
"--volume",
"/etc/openstack/clouds.yaml:/etc/openstack/clouds.yaml:rw",
"--volume",
"/etc/openstack/certs/:/etc/openstack/certs/:rw",
"--volume",
"/etc/openstack/web_config.yml:/etc/openstack/web_config.yml:rw",
"openstack-exporter:1.7.0",
"--disable-service.dns",
"--endpoint-type=public",
"--web.telemetry-path=/metrics",
"--disable-deprecated-metrics",
"--os-client-config="/etc/openstack/clouds.yaml"",
"openstack"
], i just add web_conf_file.yam as volume put i don't have any option --web.config.file= .
Thank u a lot i'm gonna change this.

[Jonher937]

That should have resolved the issue for you, have you had a chance to try it?

[noor47-pixel]

Hello, not totally,

the certificate I was given is issued by my company's CA, but the component doesn't accept the certificate. I don't know if I need to configure it when I launch the container.

Best regards

[noor47-pixel]

i got thi error ts=2024-08-07T15:04:07.860Z caller=stdlib.go:105 level=error component=web caller="http: TLS handshake error from 10.89.0.1:52678" msg="remote error: tls: bad certificate"

[Jonher937]

Bad certificate usually means you either don't trust it, or trying to connect on a name that is not present in that certificate.
Just as I did above, you can try to curl the endpoint and see what certificate and name is presented. Does the exporter stat "TLS is enabled" now ?

Try: curl -vk https://<name-in-cert>:8444 that will ignore SSL trust and print verbose like my example above.

[noor47-pixel]

Yes it's enabled now i see on logs outputs.

i tried command: curl -vk https://dns:8444 then i got this.
Everyting is OK with the name ( DNS) about the server. And i give also ca into my web_config.yml so I can't see where's the problem is comming from .
there is output from curl result:

• Rebuilt URL to: https://dns:8444/
• Trying 55.x.x.x...
• TCP_NODELAY set
• connect to 55.x.x.x port 8444 failed: Connexion refusée
• Failed to connect to dns port 8444: Connexion refusée
• Closing connection 0
  curl: (7) Failed to connect to 55.x.x.x port 8444: Connexion refusée

[Jonher937]

Connection refused. Looks like you can't reach the server, such as a firewalling issue.

[noor47-pixel]

but i execute this command on the server itself

[Jonher937]

That does not necessarily allow the traffic. Start with checking what the process binds to IP wise: sudo ss -tlpn | grep 8444
Then try curl -kv https://127.0.0.1:8444 for example to verify you can reach the service on localhost (if it binds there) and that the cert is served.

[Dmitry-Eremeev]

@noor47-pixel
Please specify these settings:
client_auth_type: RequireAndVerifyClientCert
client_ca_file: <YOUR_CUSTOM_CA_CERT>

https://github.com/prometheus/exporter-toolkit/blob/master/docs/web-configuration.md