FIPS broken for Ruby on Windows with OpenSSL 3.x

[tpowell-progress]

Description

This is to document the current state of FIPS with OpenSSL 3.x for Ruby on Windows.

In validation of the OpenSSL 3 upgrade, I noticed that FIPS mode fails to reject MD5 hashes. Upon further inspection, it appears that OpenSSL was routing through the default provider and any attempt to ensure that the default provider is excluded breaks SSL functionality.

In researching the problem, I found openssl#603, which indicates that OpenSSL 3 FIPS support is still an open issue. The issue seems to indicate that only testing remains; however, the PR to fix OpenSSL::PKey.read that cannot parse PKey in the FIPS mode seems to not prevent the error on a Windows installation of Ruby. (I've installed both the 3.2.0 gem and pulled from latest master as well as confirming the fix code is actually installed.)

Chef Version

Upcoming release 18.5

Platform Version

Windows with FIPS mode

Replication Case

    & "$embedded_bin_dir/openssl.exe list -providers" # if default provider is included, the below fails to reject MD5, else it errors with PKey error

    & $embedded_bin_dir/ruby.exe -v -e "require 'openssl'; begin;OpenSSL::Digest::MD5.new('hi');rescue OpenSSL::Digest::DigestError => e;raise;rescue => e;puts 'Unexpected error:';puts e.inspect;end"

Client Output

c:/ruby31/lib/ruby/gems/3.1.0/gems/openssl-3.2.0/lib/openssl/pkey.rb:132:in `initialize': could not parse pkey (OpenS
SL::PKey::DHError)
        from c:/ruby31/lib/ruby/gems/3.1.0/gems/openssl-3.2.0/lib/openssl/pkey.rb:132:in `new'
        from c:/ruby31/lib/ruby/gems/3.1.0/gems/openssl-3.2.0/lib/openssl/pkey.rb:132:in `new'
        from c:/ruby31/lib/ruby/gems/3.1.0/gems/openssl-3.2.0/lib/openssl/ssl.rb:36:in `<class:SSLContext>'
        from c:/ruby31/lib/ruby/gems/3.1.0/gems/openssl-3.2.0/lib/openssl/ssl.rb:23:in `<module:SSL>'
        from c:/ruby31/lib/ruby/gems/3.1.0/gems/openssl-3.2.0/lib/openssl/ssl.rb:22:in `<module:OpenSSL>'
        from c:/ruby31/lib/ruby/gems/3.1.0/gems/openssl-3.2.0/lib/openssl/ssl.rb:21:in `<top (required)>'
        from c:/ruby31/lib/ruby/gems/3.1.0/gems/openssl-3.2.0/lib/openssl.rb:21:in `require_relative'
        from c:/ruby31/lib/ruby/gems/3.1.0/gems/openssl-3.2.0/lib/openssl.rb:21:in `<top (required)>'
        from <internal:c:/ruby31/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
        from <internal:c:/ruby31/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
        from -e:1:in `<main>'

Stacktrace