TLS Server Error – Fatal Bad Certificate Alert

[Abhijiththinkpalm]

We are encountering a TLS error in our EMQX server. The server is receiving a fatal Bad Certificate alert from the client when in the wait_cert state. Below is the relevant log snippet:

TLS server: In state wait_cert at ssl_handshake.erl:2148 generated SERVER ALERT: Fatal - Bad Certificate, - invalid_signature
2025-03-24T14:22:17.158969+05:30 [notice] supervisor: {esockd_connection_sup,<0.4064.0>}, errorContext: ssl_error, reason: {tls_alert,{bad_certificate,"TLS server: In state wait_cert at ssl_handshake.erl:2148 generated SERVER ALERT: Fatal - Bad Certificate\n invalid_signature"}}, offender: [{pid,<0.4064.0>},{name,connection},{mfargs,{emqx_connection,start_link,[#{listener => {ssl,default},limiter => #{connection => #{initial => 0,burst => 0,rate => infinity}},enable_authn => true,zone => default}]}}]

Findings:

The certificates are correctly generated and work without issues in Mosquitto.
Standard RSA certificates work fine with EMQX.
The issue arises specifically when using RSASSA-PSS certificates.

Steps Taken:

Verified that the certificate chain is correct.
Confirmed that the EMQX TLS configuration is correct.
Successfully tested the RSASSA-PSS certificates with Mosquitto, confirming they are functional.
Standard RSA certificates work fine with the same configuration.

We need support in understanding why EMQX rejects RSASSA-PSS certificates while they work fine with Mosquitto. Does EMQX have specific limitations or configurations required for RSASSA-PSS certificates?

Thank you

[qzhuyan]

hi,

which EMQX version ?

I assume you are using TLS1.2?

v5.8.4(emqx@127.0.0.1)6> ssl:signature_algs(default, 'tlsv1.2').
[{sha512,ecdsa},
 rsa_pss_pss_sha512,rsa_pss_rsae_sha512,
 {sha512,rsa},
 {sha384,ecdsa},
 rsa_pss_pss_sha384,rsa_pss_rsae_sha384,
 {sha384,rsa},
 {sha256,ecdsa},
 rsa_pss_pss_sha256,rsa_pss_rsae_sha256,
 {sha256,rsa}]

[zmstone]

Maybe a bug in ssl lib, created an issue for OTP team: erlang/otp#9632

[Abhijiththinkpalm]

hi,

which EMQX version ?

I assume you are using TLS1.2?

v5.8.4(emqx@127.0.0.1)6> ssl:signature_algs(default, 'tlsv1.2').
[{sha512,ecdsa},
 rsa_pss_pss_sha512,rsa_pss_rsae_sha512,
 {sha512,rsa},
 {sha384,ecdsa},
 rsa_pss_pss_sha384,rsa_pss_rsae_sha384,
 {sha384,rsa},
 {sha256,ecdsa},
 rsa_pss_pss_sha256,rsa_pss_rsae_sha256,
 {sha256,rsa}]

using tlsv1.3 and tlsv1.2 in configuration
v5.8.5(emqx@127.0.0.1)8> ssl:signature_algs(default, 'tlsv1.3').
[eddsa_ed25519,eddsa_ed448,ecdsa_secp521r1_sha512,
ecdsa_secp384r1_sha384,ecdsa_secp256r1_sha256,
rsa_pss_pss_sha512,rsa_pss_pss_sha384,rsa_pss_pss_sha256,
rsa_pss_rsae_sha512,rsa_pss_rsae_sha384,rsa_pss_rsae_sha256,
rsa_pkcs1_sha512,rsa_pkcs1_sha384,rsa_pkcs1_sha256,
{sha512,ecdsa},
{sha384,ecdsa},
{sha256,ecdsa}]

[Abhijiththinkpalm]

Also, I noticed that it works with emqx version 4.4.

[zmstone]

Thank your for the update @Abhijiththinkpalm.
We'll follow up with the issue reported to OTP team.

[Abhijiththinkpalm]

After implementing EMQX in a production setting, we became aware of the issue. So it would be greatly appreciated if you could resolve the issue as quickly as possible.

[Abhijiththinkpalm]

@zmstone any updates?

[zmstone]

Sorry, No update from OTP team yet.

[zmstone]

Hi. OTP team accepted the issue report.
erlang/otp#9632

We will try to upgrade OTP version after the fix is released.

[zmstone]

Hi Again @Abhijiththinkpalm

It would be nice if you can provide the certificates chain (no need for private keys) for my inspection.
Just to make sure you are indeed hitting the exact bug erlang/otp#9632

[Abhijiththinkpalm]

Hi @zmstone
Pls see the attached certificates that i used for testing.

ca.pem

-----BEGIN CERTIFICATE-----
MIIDiTCCAj2gAwIBAgIUfu8TTrQyWMs+T/pu/NY/RYTBxywwQQYJKoZIhvcNAQEK
MDSgDzANBglghkgBZQMEAgEFAKEcMBoGCSqGSIb3DQEBCDANBglghkgBZQMEAgEF
AKIDAgEgMBgxFjAUBgNVBAMMDUNoaXJwU3RhY2sgQ0EwHhcNMjUwMzI0MDY0OTU4
WhcNMzUwMzIyMDY0OTU4WjAYMRYwFAYDVQQDDA1DaGlycFN0YWNrIENBMIIBIjAN
BgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAqKZD5nmxTw/EUNQgPFyKR3fx7EKO
wmnE8bVUCJOPT/RmKfA1h6YgvEGb8U/A0C9S9M7WKkO0I9DTVg+kJMYmsKd0WChU
CJdTZp7DsdBeDENDfo7X/+iFuIVnsbZJWfo5bYedN8MGr7kBRPEN/PoykDW63Usu
9PPD3p0HfPQpdQ4nNtXfhLh7kOgj3Nd7FCTW3BTnJoZPy5NE5s8Mt1z5Q54zeh54
itZKiuROTZvtw0Rr+Qa1B27fzMWarQtq0DxbvI0TibGl/mBQ1fvDP2CMKO3kkwxe
A8eoGrmAX/lJeYs4oaVz616YdJY/rkk2SSjGtEScHqayV3KyKkGoZ+9OAQIDAQAB
o2MwYTAdBgNVHQ4EFgQUAr70tpH4t6uLlz9l56tMfh5qXa8wHwYDVR0jBBgwFoAU
Ar70tpH4t6uLlz9l56tMfh5qXa8wDgYDVR0PAQH/BAQDAgKkMA8GA1UdEwEB/wQF
MAMBAf8wQQYJKoZIhvcNAQEKMDSgDzANBglghkgBZQMEAgEFAKEcMBoGCSqGSIb3
DQEBCDANBglghkgBZQMEAgEFAKIDAgEgA4IBAQAzNHbjHXjvDvjfYyEpkKbCcq4a
JGJohtwMQh0eizsgtKLyY4oIeQiC1P+k+NAP630yncgzV0n72y6O1uNxt0tEeNsO
XwW6lpdF/WIn7ItcOa5H56gC8wgTQQBHD8UELQC2qRK6VpisWoWUu3tJocLTzh8D
rj/Exdjll6ZsolbKH0XBfWJrVpeyVyCiIR2ZZfFa6J9gfEAkML/LBOs1Hq68IbsG
MdMGeutgswNc0RN/Hhc9+sT8LU9TAfHtjnT7h/JRDZfKCXwtKg+f2CCCerIQeezi
VVCcDvqZz3rI+VHEu3OpIbYAMOjhgqqGG2Q+93izeOs2oNYkWnofn0wGuWXS
-----END CERTIFICATE-----

server.pem

-----BEGIN CERTIFICATE-----
MIIDiTCCAj2gAwIBAgIUWJ9SjLtB19fh15V40QrOQ+XCQ9wwQQYJKoZIhvcNAQEK
MDSgDzANBglghkgBZQMEAgEFAKEcMBoGCSqGSIb3DQEBCDANBglghkgBZQMEAgEF
AKIDAgEgMBgxFjAUBgNVBAMMDUNoaXJwU3RhY2sgQ0EwHhcNMjUwMzI0MDY1MTA0
WhcNMzUwMzIyMDY1MTA0WjAUMRIwEAYDVQQDDAlsb2NhbGhvc3QwggEiMA0GCSqG
SIb3DQEBAQUAA4IBDwAwggEKAoIBAQC/Sf0KKJRWKobYsRQvIkDCNOaaQCFLf5zB
b6g9YHpYTtGzuk0HgLrM7I1VXQaqGSLPyLt47+FSfJKiu5D8Y/6jFUZDaTS7x9c+
QhR1Jtv40uyKEVger5l8YoKhAwhAYV/xR2ilZELiEN4lloHqG1YRCsGA5Mvqa0Zj
6viBjjsXx6kYropAbJ0Ew8rpcFffyrAj1o7/jK7LtF9wVwS/N5qDUVbbqlHN3NOX
a6+hecs9r8SjY01RgN/HuKKa4JKjrl4x4CC0a+oePbB2yFerypUvIkmBzTtSKa/i
kD4VA615P/M41JVSjDrgpCAZbFcFW1auXrDKXzcaQj5/H4yy3HDhAgMBAAGjZzBl
MA4GA1UdDwEB/wQEAwIFoDATBgNVHSUEDDAKBggrBgEFBQcDATAdBgNVHQ4EFgQU
I3K8w+aFR8pmd1d/ny/qUgbtGmkwHwYDVR0jBBgwFoAUAr70tpH4t6uLlz9l56tM
fh5qXa8wQQYJKoZIhvcNAQEKMDSgDzANBglghkgBZQMEAgEFAKEcMBoGCSqGSIb3
DQEBCDANBglghkgBZQMEAgEFAKIDAgEgA4IBAQCIZLS4xXSXDQf88UuAMQUXmUFy
b6mn0ngVu+82/8fq4eEd5FU5u5xZAECqICAdOlYMHJyE6X9UCRitJUbptA9m5bgA
r/m6QRRZACaaWAkTkYoJkl0+R4NIN08n137ziUJHBc4kPccjH4qEQJ30vbbCVx89
+y5ObfbSpLUBU6bfEMjs/C2EV6qwz2PiiFWKRVeQ3AzdZ+hSqObmu5NCWyzAp2jj
PfFQ69W2rK1ZzBy3WwgQ0SoHcHXU2NeR/W09S6NLNkyyCOuGO29cm7PrPyvPnoFy
VxI29TfJpIOZ7W2FSlCGkojGwGyRgewigMAboHvIelXktf4TmarWJrbttHUJ
-----END CERTIFICATE-----

client.pem

-----BEGIN CERTIFICATE-----
MIIDizCCAj+gAwIBAgIUWJ9SjLtB19fh15V40QrOQ+XCQ90wQQYJKoZIhvcNAQEK
MDSgDzANBglghkgBZQMEAgEFAKEcMBoGCSqGSIb3DQEBCDANBglghkgBZQMEAgEF
AKIDAgEgMBgxFjAUBgNVBAMMDUNoaXJwU3RhY2sgQ0EwHhcNMjUwMzI0MDY1MTQ3
WhcNMzUwMzIyMDY1MTQ3WjAWMRQwEgYDVQQDDAttcXR0LWNsaWVudDCCASIwDQYJ
KoZIhvcNAQEBBQADggEPADCCAQoCggEBAJ45MFoBkZ5vHoOUc4iuuplE4QZ2vHiE
N9AZiu+g6BXoXXXgUTqowQElVnQUKxsD0DI00h9r0yUc7MrdDi3ww67+YVhFYKXP
QfZyTSeUtnww5kJJYkKVgtaW20twJpEI88+PiWgj0F5YkV/1zF+l5BOHEVNb3NnV
/JCK63Lf4mpJ1mAKk2ZBbAq2hBmd6mu0fc5mU1Sq6vjf6gTPNi0v5lSGy6+Q1ZS8
dEE4S/qamITVV9b7USpR4+7itamtabJ0R8AU9lHDiZKDXgdTpKURDVWwxZrQQYGX
N1M6G/tyMSBuafWjFLZqZ+fjxkLY4J5ioSDWwusXGL9XgkKgHwIIfyECAwEAAaNn
MGUwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQMMAoGCCsGAQUFBwMCMB0GA1UdDgQW
BBRkPxi0hwtK4FTsXMHM9PXLsoasTjAfBgNVHSMEGDAWgBQCvvS2kfi3q4uXP2Xn
q0x+HmpdrzBBBgkqhkiG9w0BAQowNKAPMA0GCWCGSAFlAwQCAQUAoRwwGgYJKoZI
hvcNAQEIMA0GCWCGSAFlAwQCAQUAogMCASADggEBACoQ2kWU6YfG/wHigzpjagLm
Rh1fh4+CMakNxB/fGS9eivW2BgpEO/wOO+IKvVHUQ/qOnMpr7kd4Vi1hYneuGfJw
Sx/PmNovO/9X80EfovwirAUe3bJkuGkWlOXFt8dWXFO8gZ3H4HaZ04dSrpFO3pYo
0+FzSPsSdwwLjEPx69We7+aYT89BIn92m8yAqhrMGhtvVpL5J7LIBBoYzK1CC8Gr
WuLB4WTc4KOdLVNXve3PszcWPdHIt7GWdq8rMJDM0OWTsmeMngtP7j5KZkdWpQ+Q
5EcPQ6oNbNQSnd2EUoR2CvD+A6nOuuhV2SgCRPBGxNQ7EPiwgw0SyTTdHtGCvUo=
-----END CERTIFICATE-----

[zmstone]

Thank you. @Abhijiththinkpalm
It confirms my findings.

[Abhijiththinkpalm]

Hi @zmstone
When is the problem going to be resolved?
since erlang/otp added to [OTP-28.1]
can u give a estimate from your side

[zmstone]

Hi @Abhijiththinkpalm

The fix is likely going to be back-ported to OTP 27, after that, we can upgrade in the very next release.
I do not know when this fix will be available from OTP 27, but we release patch versions every 1-2 months.

If the fix will only be available in OTP 28, it's going to take a bit longer for us to upgrade.
OTP version upgrades usually happens in EMQX's minor releases. e.g. 5.11 or 5.12

[zmstone]

It's fixed in https://github.com/erlang/otp/releases/tag/OTP-26.2.5.14
and https://github.com/erlang/otp/releases/tag/OTP-27.3.4.2
The fix will be available in EMQX 6.0 and maybe 5.10

[zmstone]

Will be a part of 5.8.7 5.8.8 release: #15581

Fixes for 5.9.2 and 5.10.1 will follow.

[zmstone]

Does the emqx5.8 docker images have this fix?

yes, please wait for the next release, 5.8.8. (sorry, I previously thought 5.8.7 was the next release).

[Abhijiththinkpalm]

Its alright.
please inform after release.

[Abhijiththinkpalm]

Hi @zmstone

any updates?

[keynslug]

Hi @Abhijiththinkpalm, I believe https://github.com/emqx/emqx/releases/tag/v5.8.8 has the fix in question, see the release notes under Security section.

[Abhijiththinkpalm]

Hi @zmstone or @keynslug
i don't think the issue is resolved

pls see the log from emqx when i connect to using rss pss certificate

2025-09-22T18:56:49.012951+05:30 [notice] TLS server: In state start at tls_server_connection_1_3.erl:698 generated SERVER ALERT: Fatal - Handshake Failure, - unable_to_supply_acceptable_cert
2025-09-22T18:56:49.013551+05:30 [notice] supervisor: {esockd_connection_sup,<0.4315.0>}, errorContext: ssl_error, reason: {tls_alert,{handshake_failure,"TLS server: In state start at tls_server_connection_1_3.erl:698 generated SERVER ALERT: Fatal - Handshake Failure\n unable_to_supply_acceptable_cert"}}, offender: [{pid,<0.4315.0>},{name,connection},{mfargs,{emqx_connection,start_link,[#{listener => {ssl,default},limiter => #{connection => #{initial => 0,burst => 0,rate => infinity}},enable_authn => true,zone => default}]}}]

[zmstone]

It's not Bad Certificate\n invalid_signature" this time.

unable_to_supply_acceptable_cert is an undocumented error,
I had a look into the source code,
the error means: the server couldn’t find any of its configured certificate/key pairs that were acceptable to the client’s TLS 1.3 handshake requirements (signature algorithms + certificate authorities).

maybe:

• server cert uses unsupported signature algorithm
• wrong CA ? maybe server certificate is not issued by any of the CA acceptable by client

You can maybe enable debug level logging on the ssl listener (and set EMQX log to debug too), then share the debug logs.

listeners {
  ssl {
    default {
      bind = "0.0.0.0:8883"
      ssl_options {
        cacertfile = "${EMQX_ETC_DIR}/certs/cacert.pem"
        certfile = "${EMQX_ETC_DIR}/certs/cert.pem"
        keyfile = "${EMQX_ETC_DIR}/certs/key.pem"
        log_level = debug #<-----------------------------------------here
      }

Or share client/server certificates and keys which can reproduce this issue.

[Abhijiththinkpalm]

Hi @zmstone

The issue was on my side — I had generated the wrong type of certificate. Your solution works perfectly now.

Thank you

[nicholash-quest]

Will be a part of 5.8.7 5.8.8 release: #15581

Fixes for 5.9.2 and 5.10.1 will follow.

Hi, can you validate that this is in 5.10.1?
I see this or a very similiar error in 5.10.1 (TLS server: In state certify at ssl_handshake.erl:2192 generated SERVER ALERT: Fatal - Bad Certificate, - invalid_signature).