TLS handshake fails(radsecproxy) with OpenSSL 3.0.8 but succeeds with OpenSSL 1.1.1t

[SatheshkA]

Radsecproxy application is unable to establish secure connection with the RADSEC server. The TLS handschake fails with failed: error:0A000086:SSL routines::certificate verify failed . However with OpenSSL 1.1.1t the TLS handshake succeeds, without changing anything.

Radsecproxy version : v1.9.1

[t8m]

To be able to help we would have to see what are the certificates the server is using including the whole chain to the root CA certificate you have in the certificate trust store.

Also could you please add more information about the platform you're using? Such as where are the OpenSSL 3.0.8 and 1.1.1t coming from - whether you built them yourself, what was the build configuration, etc.

[SatheshkA]

Platform: openwrt
In openssl 1.1.1t root CA certificate issue was not happen, after upgrading openssl to 3.0.8 we could observe these issue.

[t8m]

I am sorry but this is not enough information to be able to help you. Either there is some problem with the certificate chain, or the root CA is not present in the trust store for the 3.0.8 version, which could happen if the build configuration for the 1.1.1t and 3.0.8 is different, i.e. has different OPENSSLDIR.

[kroeckx]

Some other reason might be that that one of the certificates now has a too small key, or is signed using sha1.