Inconsistent behavior building on Linux/MacOS against OpenSSL 3.x

[udkyo]

Describe the bug
Building from source with fips enabled using --with-ssl=[OpenSSL 3.x dir] on macos fails at Checking for FIPS_mode_set()... no

An identical build on Linux completes successfully (Checking for FIPS_mode_set()... yes) which is confusing, as it seems FIPS_mode_set doesn't actually exist in OpenSSL 3.x.

To Reproduce

mkdir /tmp/erlang-test
cd /tmp/erlang-test

git clone https://github.com/openssl/openssl
pushd openssl
./Configure enable-fips --prefix=/tmp/erlang-test/openssl-3.x
make && make install
popd

git clone https://github.com/erlang/otp
pushd otp
./configure --enable-fips --with-ssl=/tmp/erlang-test/openssl-3.x

Expected behavior
I would expect to see an OpenSSL 3.x friendly check, rather than a FIPS_mode_set() check which seems to only be valid when using OpenSSL 1.x.

As the FIPS_mode_set check is there however, I also expect current Linux builds --with-ssling OpenSSL 3.x to fail, which does not match my experience.

Affected versions
I encountered this while attempting to build 26.2 with FIPS enabled, but it looks like anything newer is the same. I'm not sure which older versions this behaviour is present in.

[sverker]

@udkyo Could you try #8277 where I have removed the check for FIPS_mode_set which was incorrect in multiple ways. It's currently based on OTP-26.2.3.

[udkyo]

Looks good @sverker. Using that PR on OTP-26.2.3 I'm able to build on MacOS + Linux and enable FIPS on each. Thank you very much!

[sverker]

#8277 merged. Scheduled for next 25, 26 and 27.0 release.