Disable ED25519 and ED448 in FIPS mode on openssl3

[pemensik]

Both crypto functions are not allowed by FIPS 140-3. Use openssl 3.0
function to check FIPS mode presence and use it to make those algorithms
unsupported.

Would allow flawless fallback to insecure results for ED25519, ED448 and also SHA-1 algorithms.

[pemensik]

Reference: https://bugzilla.redhat.com/show_bug.cgi?id=2079548

[pemensik]

What is FIPS mode for and how can it be used is written in RHEL documentation: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/security_hardening/assembly_installing-a-rhel-8-system-with-fips-mode-enabled_security-hardening

[pemensik]

Any comments for this change?

[gthess]

Although the code LGTM, I find the change very implicit i.e., if "just fips" is set, disable those algorithms.

[pemensik]

It works around the resolution failure in FIPS mode:

# rpm -q unbound
unbound-1.13.1-13.el9.x86_64
# fips-mode-setup --check && unbound-host -rvD secure.d2a16n3.rootcanary.net
Installation of FIPS modules is not completed.
FIPS mode is disabled.
secure.d2a16n3.rootcanary.net has address 145.97.20.20 (secure)
secure.d2a16n3.rootcanary.net has IPv6 address 2001:610:188:408::20 (secure)
secure.d2a16n3.rootcanary.net has no mail handler record (secure)
# fips-mode-setup --enable && reboot
# fips-mode-setup --check && unbound-host -rvD secure.d2a16n3.rootcanary.net
FIPS mode is enabled.
secure.d2a16n3.rootcanary.net has address 145.97.20.20 (BOGUS (security failure))
validation failure <secure.d2a16n3.rootcanary.net. A IN>: use of key for crypto failed from 10.x.x.1 for key d2a16n3.rootcanary.net. while building chain of trust
secure.d2a16n3.rootcanary.net has IPv6 address 2001:610:188:408::20 (BOGUS (security failure))
validation failure <secure.d2a16n3.rootcanary.net. AAAA IN>: key for validation d2a16n3.rootcanary.net. is marked as invalid because of a previous validation failure <secure.d2a16n3.rootcanary.net. A IN>: use of key for crypto failed from 10.2.32.1 for key d2a16n3.rootcanary.net. while building chain of trust
secure.d2a16n3.rootcanary.net has no mail handler record (BOGUS (security failure))
validation failure <secure.d2a16n3.rootcanary.net. MX IN>: key for validation d2a16n3.rootcanary.net. is marked as invalid because of a previous validation failure <secure.d2a16n3.rootcanary.net. A IN>: use of key for crypto failed from 10.2.32.1 for key d2a16n3.rootcanary.net. while building chain of trust

After this change the result in FIPS mode would switch just to insecure, but still to successful resolution. Of course unbound itself would be affected too.

[pemensik]

I expect FIPS mode is not enabled in openssl unless there is a reason for it to be. This might change once those algorithms are accepted by FIPS 140-3. It is expected they might and would be, but not clear when. This change can be reverted once they are acceptable.

[mnordhoff]

Making security worse for "compliance" is sad.