OpenSSL 1.1.1 compatibility

[voxik]

OpenSSL 1.1.1 have landed in Fedora Rawhide:

$ rpm -q openssl-devel
openssl-devel-1.1.1-0.pre9.fc29.x86_64

Running Ruby 2.5.1 test suite, I observe the following error:

  1) Failure:
TestGemRemoteFetcher#test_do_not_allow_invalid_client_cert_auth_connection [/builddir/build/BUILD/ruby-2.5.1/test/rubygems/test_gem_remote_fetcher.rb:845]:
[Gem::RemoteFetcher::FetchError] exception expected, not
Class: <OpenSSL::SSL::SSLError>
Message: <"SSL_read: tlsv1 alert decrypt error">
---Backtrace---
/builddir/build/BUILD/ruby-2.5.1/.ext/common/openssl/buffering.rb:182:in `sysread_nonblock'
/builddir/build/BUILD/ruby-2.5.1/.ext/common/openssl/buffering.rb:182:in `read_nonblock'
/builddir/build/BUILD/ruby-2.5.1/lib/net/protocol.rb:175:in `rbuf_fill'
/builddir/build/BUILD/ruby-2.5.1/lib/net/protocol.rb:157:in `readuntil'
/builddir/build/BUILD/ruby-2.5.1/lib/net/protocol.rb:167:in `readline'
/builddir/build/BUILD/ruby-2.5.1/lib/net/http/response.rb:40:in `read_status_line'
/builddir/build/BUILD/ruby-2.5.1/lib/net/http/response.rb:29:in `read_new'
/builddir/build/BUILD/ruby-2.5.1/lib/net/http.rb:1497:in `block in transport_request'
/builddir/build/BUILD/ruby-2.5.1/lib/net/http.rb:1494:in `catch'
/builddir/build/BUILD/ruby-2.5.1/lib/net/http.rb:1494:in `transport_request'
/builddir/build/BUILD/ruby-2.5.1/lib/net/http.rb:1467:in `request'
/builddir/build/BUILD/ruby-2.5.1/lib/rubygems/request.rb:221:in `perform_request'
/builddir/build/BUILD/ruby-2.5.1/lib/rubygems/request.rb:156:in `fetch'
/builddir/build/BUILD/ruby-2.5.1/lib/rubygems/remote_fetcher.rb:368:in `request'
/builddir/build/BUILD/ruby-2.5.1/lib/rubygems/remote_fetcher.rb:251:in `fetch_http'
/builddir/build/BUILD/ruby-2.5.1/lib/rubygems/remote_fetcher.rb:292:in `fetch_path'
/builddir/build/BUILD/ruby-2.5.1/test/rubygems/test_gem_remote_fetcher.rb:846:in `block (2 levels) in test_do_not_allow_invalid_client_cert_auth_connection'
---------------

Originally, I reported the test suite issues here [1] and all except this one were already fixed. Just FTR, here is the output of your SSL check script:

$ make runruby TESTRUN_SCRIPT="check.rb"
./miniruby -I./lib -I. -I.ext/common  ./tool/runruby.rb --extout=.ext  -- --disable-gems check.rb
Here's your Ruby and OpenSSL environment:

Ruby:           2.5.1p57 (2018-03-29 revision 63029) [x86_64-linux]
RubyGems:       2.7.6
Compiled with:  OpenSSL 1.1.1-pre9 (beta) FIPS 21 Aug 2018
Loaded version: OpenSSL 1.1.1-pre9 (beta) FIPS 21 Aug 2018
SSL_CERT_FILE:  /etc/pki/tls/cert.pem
SSL_CERT_DIR:   /etc/pki/tls/certs

With that out of the way, let's see if you can connect to rubygems.org...

Bundler connection to rubygems.org:       failed  ❌  (uninitialized constant Bundler)
RubyGems connection to rubygems.org:      success ✅
Ruby net/http connection to rubygems.org: success ✅

Although your Ruby installation and RubyGems can both connect to rubygems.org, Bundler is having trouble. The most likely way to fix this is to upgrade Bundler by running `gem install bundler`. Run this script again after doing that to make sure everything is all set. If you're still having trouble, check out the troubleshooting guide at http://ruby.to/ssl-check-failed 📦
check.rb:136: warning: constant OpenSSL::SSL::SSLContext::METHODS is deprecated

[MSP-Greg]

@voxik

FYI, using ruby 2.6.0dev (2018-08-27 trunk 64554) [x64-mingw32] built using OpenSSL master as of 24-Aug, test-all passed. Also, check.rb passed, some output below:

Ruby:           2.6.0p-1 (2018-08-27 revision 64554) [x64-mingw32]
RubyGems:       3.0.0.beta1
Bundler:        1.16.4
Compiled with:  OpenSSL 1.1.1-pre10-dev  24 Aug 2018
Loaded version: OpenSSL 1.1.1-pre10-dev  24 Aug 2018

I noticed the message <"SSL_read: tlsv1 alert decrypt error">. Was it built with TLSv1 support?

Finally, 2.5.1 uses Ruby OpenSSL 2.1.0. Have you tried installing the openssl gem, which is 2.1.1? If that worked...

[MSP-Greg]

I probably should have let everyone here know...

The Appveyor trunk build has been using OpenSSL 1.1.1 since ruby 2.6.0dev (2018-08-19 trunk 64471) [x64-mingw32]

For a couple of weeks, I tested locally using OpenSSL 1.1.1, building & testing trunk, and also building & testing Puma & EventMachine. All were stable, so I added it to trunk.

I'm updating the OpenSSL package once or twice a week until release. I'll see issues right away in the RubyGems tests in the Ruby test-all suite, but if any crop up here and I don't notice them, please ping me.

Thanks, Greg

[MSP-Greg]

@voxik,

I did a "don't try this at home", and swapped OpenSSL dll's from 1.1.0 to 1.1.1. Using 2.5.1, all tests in this repo passed.

I tried it with both the default 2.5.1 OpenSSL (2.1.0) and also with a gem built yesterday from Ruby OpenSSL master (2.1.1+?). Both were built using 1.1.0, but both work with the 1.1.1 dll's.

I'm not sure what might be causing the different results. I haven't tried building 2.5.1 from scratch.

FYI, the 'SSL check script' also 'passed'. As mentioned above, below is OpenSSL version info:

Compiled with:  OpenSSL 1.1.0h  27 Mar 2018
Loaded version: OpenSSL 1.1.1-pre10-dev  24 Aug 2018

[voxik]

Probably different configuration? These are the settings from Fedora Rawhide:

$ cat /etc/crypto-policies/back-ends/openssl.config 
@SECLEVEL=1:kEECDH:kRSA:kEDH:-aDSS:!EXP:-3DES:!DES:!RC4:!RC2:!IDEA:-SEED:!eNULL:!aNULL:!MD5:-SHA384:-CAMELLIA:!SSLv2:!ADH

$ cat /etc/crypto-policies/back-ends/opensslcnf.config 
CipherString = @SECLEVEL=1:kEECDH:kRSA:kEDH:-aDSS:!EXP:-3DES:!DES:!RC4:!RC2:!IDEA:-SEED:!eNULL:!aNULL:!MD5:-SHA384:-CAMELLIA:!SSLv2:!ADH
Ciphersuites = TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256:TLS_AES_128_CCM_SHA256
MinProtocol = TLSv1

I am not OpenSSL expert, so I dunno how they are actually applied, but I guess that OPENSSL_CONF env var could be the env variable to provide configuration overrides.

[MSP-Greg]

I am not OpenSSL expert

For any of the deep SSL stuff I have google open and probably some man pages. I thought that there aren't a lot of restrictions that can be set thru the OpenSSL conf file. But, if that's the case, why would they affect Ruby? So, I'm confused...

BTW, what is the output from:

ruby -ropenssl -e "puts OpenSSL::Config::DEFAULT_CONFIG_FILE || 'dflt missing', ENV['OPENSSL_CONF'] || 'ENV missing'"

[voxik]

$ ruby -ropenssl -e "puts OpenSSL::Config::DEFAULT_CONFIG_FILE || 'dflt missing', ENV['OPENSSL_CONF'] || 'ENV missing'"
/etc/pki/tls/openssl.cnf
ENV missing

$ cat /etc/pki/tls/openssl.cnf
#
# OpenSSL example configuration file.
# This is mostly being used for generation of certificate requests.
#

# Note that you can include other files from the main configuration
# file using the .include directive.
#.include filename

# This definition stops the following lines choking if HOME isn't
# defined.
HOME			= .
RANDFILE		= $ENV::HOME/.rnd

# Extra OBJECT IDENTIFIER info:
#oid_file		= $ENV::HOME/.oid
oid_section		= new_oids

# To use this configuration file with the "-extfile" option of the
# "openssl x509" utility, name here the section containing the
# X.509v3 extensions to use:
# extensions		= 
# (Alternatively, use a configuration file that has only
# X.509v3 extensions in its main [= default] section.)

# Load default TLS policy configuration

openssl_conf = default_modules

[ default_modules ]

ssl_conf = ssl_module

[ ssl_module ]

system_default = crypto_policy

[ crypto_policy ]

.include /etc/crypto-policies/back-ends/opensslcnf.config

[ new_oids ]

# We can add new OIDs in here for use by 'ca', 'req' and 'ts'.
# Add a simple OID like this:
# testoid1=1.2.3.4
# Or use config file substitution like this:
# testoid2=${testoid1}.5.6

# Policies used by the TSA examples.
tsa_policy1 = 1.2.3.4.1
tsa_policy2 = 1.2.3.4.5.6
tsa_policy3 = 1.2.3.4.5.7

####################################################################
[ ca ]
default_ca	= CA_default		# The default ca section

####################################################################
[ CA_default ]

dir		= /etc/pki/CA		# Where everything is kept
certs		= $dir/certs		# Where the issued certs are kept
crl_dir		= $dir/crl		# Where the issued crl are kept
database	= $dir/index.txt	# database index file.
#unique_subject	= no			# Set to 'no' to allow creation of
					# several certs with same subject.
new_certs_dir	= $dir/newcerts		# default place for new certs.

certificate	= $dir/cacert.pem 	# The CA certificate
serial		= $dir/serial 		# The current serial number
crlnumber	= $dir/crlnumber	# the current crl number
					# must be commented out to leave a V1 CRL
crl		= $dir/crl.pem 		# The current CRL
private_key	= $dir/private/cakey.pem# The private key
RANDFILE	= $dir/private/.rand	# private random number file

x509_extensions	= usr_cert		# The extensions to add to the cert

# Comment out the following two lines for the "traditional"
# (and highly broken) format.
name_opt 	= ca_default		# Subject Name options
cert_opt 	= ca_default		# Certificate field options

# Extension copying option: use with caution.
# copy_extensions = copy

# Extensions to add to a CRL. Note: Netscape communicator chokes on V2 CRLs
# so this is commented out by default to leave a V1 CRL.
# crlnumber must also be commented out to leave a V1 CRL.
# crl_extensions	= crl_ext

default_days	= 365			# how long to certify for
default_crl_days= 30			# how long before next CRL
default_md	= sha256		# use SHA-256 by default
preserve	= no			# keep passed DN ordering

# A few difference way of specifying how similar the request should look
# For type CA, the listed attributes must be the same, and the optional
# and supplied fields are just that :-)
policy		= policy_match

# For the CA policy
[ policy_match ]
countryName		= match
stateOrProvinceName	= match
organizationName	= match
organizationalUnitName	= optional
commonName		= supplied
emailAddress		= optional

# For the 'anything' policy
# At this point in time, you must list all acceptable 'object'
# types.
[ policy_anything ]
countryName		= optional
stateOrProvinceName	= optional
localityName		= optional
organizationName	= optional
organizationalUnitName	= optional
commonName		= supplied
emailAddress		= optional

####################################################################
[ req ]
default_bits		= 2048
default_md		= sha256
default_keyfile 	= privkey.pem
distinguished_name	= req_distinguished_name
attributes		= req_attributes
x509_extensions	= v3_ca	# The extensions to add to the self signed cert

# Passwords for private keys if not present they will be prompted for
# input_password = secret
# output_password = secret

# This sets a mask for permitted string types. There are several options. 
# default: PrintableString, T61String, BMPString.
# pkix	 : PrintableString, BMPString (PKIX recommendation before 2004)
# utf8only: only UTF8Strings (PKIX recommendation after 2004).
# nombstr : PrintableString, T61String (no BMPStrings or UTF8Strings).
# MASK:XXXX a literal mask value.
# WARNING: ancient versions of Netscape crash on BMPStrings or UTF8Strings.
string_mask = utf8only

# req_extensions = v3_req # The extensions to add to a certificate request

[ req_distinguished_name ]
countryName			= Country Name (2 letter code)
countryName_default		= XX
countryName_min			= 2
countryName_max			= 2

stateOrProvinceName		= State or Province Name (full name)
#stateOrProvinceName_default	= Default Province

localityName			= Locality Name (eg, city)
localityName_default		= Default City

0.organizationName		= Organization Name (eg, company)
0.organizationName_default	= Default Company Ltd

# we can do this but it is not needed normally :-)
#1.organizationName		= Second Organization Name (eg, company)
#1.organizationName_default	= World Wide Web Pty Ltd

organizationalUnitName		= Organizational Unit Name (eg, section)
#organizationalUnitName_default	=

commonName			= Common Name (eg, your name or your server\'s hostname)
commonName_max			= 64

emailAddress			= Email Address
emailAddress_max		= 64

# SET-ex3			= SET extension number 3

[ req_attributes ]
challengePassword		= A challenge password
challengePassword_min		= 4
challengePassword_max		= 20

unstructuredName		= An optional company name

[ usr_cert ]

# These extensions are added when 'ca' signs a request.

# This goes against PKIX guidelines but some CAs do it and some software
# requires this to avoid interpreting an end user certificate as a CA.

basicConstraints=CA:FALSE

# Here are some examples of the usage of nsCertType. If it is omitted
# the certificate can be used for anything *except* object signing.

# This is OK for an SSL server.
# nsCertType			= server

# For an object signing certificate this would be used.
# nsCertType = objsign

# For normal client use this is typical
# nsCertType = client, email

# and for everything including object signing:
# nsCertType = client, email, objsign

# This is typical in keyUsage for a client certificate.
# keyUsage = nonRepudiation, digitalSignature, keyEncipherment

# This will be displayed in Netscape's comment listbox.
nsComment			= "OpenSSL Generated Certificate"

# PKIX recommendations harmless if included in all certificates.
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid,issuer

# This stuff is for subjectAltName and issuerAltname.
# Import the email address.
# subjectAltName=email:copy
# An alternative to produce certificates that aren't
# deprecated according to PKIX.
# subjectAltName=email:move

# Copy subject details
# issuerAltName=issuer:copy

#nsCaRevocationUrl		= http://www.domain.dom/ca-crl.pem
#nsBaseUrl
#nsRevocationUrl
#nsRenewalUrl
#nsCaPolicyUrl
#nsSslServerName

# This is required for TSA certificates.
# extendedKeyUsage = critical,timeStamping

[ v3_req ]

# Extensions to add to a certificate request

basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment

[ v3_ca ]


# Extensions for a typical CA


# PKIX recommendation.

subjectKeyIdentifier=hash

authorityKeyIdentifier=keyid:always,issuer

basicConstraints = critical,CA:true

# Key usage: this is typical for a CA certificate. However since it will
# prevent it being used as an test self-signed certificate it is best
# left out by default.
# keyUsage = cRLSign, keyCertSign

# Some might want this also
# nsCertType = sslCA, emailCA

# Include email address in subject alt name: another PKIX recommendation
# subjectAltName=email:copy
# Copy issuer details
# issuerAltName=issuer:copy

# DER hex encoding of an extension: beware experts only!
# obj=DER:02:03
# Where 'obj' is a standard or added object
# You can even override a supported extension:
# basicConstraints= critical, DER:30:03:01:01:FF

[ crl_ext ]

# CRL extensions.
# Only issuerAltName and authorityKeyIdentifier make any sense in a CRL.

# issuerAltName=issuer:copy
authorityKeyIdentifier=keyid:always

[ proxy_cert_ext ]
# These extensions should be added when creating a proxy certificate

# This goes against PKIX guidelines but some CAs do it and some software
# requires this to avoid interpreting an end user certificate as a CA.

basicConstraints=CA:FALSE

# Here are some examples of the usage of nsCertType. If it is omitted
# the certificate can be used for anything *except* object signing.

# This is OK for an SSL server.
# nsCertType			= server

# For an object signing certificate this would be used.
# nsCertType = objsign

# For normal client use this is typical
# nsCertType = client, email

# and for everything including object signing:
# nsCertType = client, email, objsign

# This is typical in keyUsage for a client certificate.
# keyUsage = nonRepudiation, digitalSignature, keyEncipherment

# This will be displayed in Netscape's comment listbox.
nsComment			= "OpenSSL Generated Certificate"

# PKIX recommendations harmless if included in all certificates.
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid,issuer

# This stuff is for subjectAltName and issuerAltname.
# Import the email address.
# subjectAltName=email:copy
# An alternative to produce certificates that aren't
# deprecated according to PKIX.
# subjectAltName=email:move

# Copy subject details
# issuerAltName=issuer:copy

#nsCaRevocationUrl		= http://www.domain.dom/ca-crl.pem
#nsBaseUrl
#nsRevocationUrl
#nsRenewalUrl
#nsCaPolicyUrl
#nsSslServerName

# This really needs to be in place for it to be a proxy certificate.
proxyCertInfo=critical,language:id-ppl-anyLanguage,pathlen:3,policy:foo

####################################################################
[ tsa ]

default_tsa = tsa_config1	# the default TSA section

[ tsa_config1 ]

# These are used by the TSA reply generation only.
dir		= /etc/pki/CA		# TSA root directory
serial		= $dir/tsaserial	# The current serial number (mandatory)
crypto_device	= builtin		# OpenSSL engine to use for signing
signer_cert	= $dir/tsacert.pem 	# The TSA signing certificate
					# (optional)
certs		= $dir/cacert.pem	# Certificate chain to include in reply
					# (optional)
signer_key	= $dir/private/tsakey.pem # The TSA private key (optional)
signer_digest  = sha256			# Signing digest to use. (Optional)
default_policy	= tsa_policy1		# Policy if request did not specify it
					# (optional)
other_policies	= tsa_policy2, tsa_policy3	# acceptable policies (optional)
digests     = sha1, sha256, sha384, sha512  # Acceptable message digests (mandatory)
accuracy	= secs:1, millisecs:500, microsecs:100	# (optional)
clock_precision_digits  = 0	# number of digits after dot. (optional)
ordering		= yes	# Is ordering defined for timestamps?
				# (optional, default: no)
tsa_name		= yes	# Must the TSA name be included in the reply?
				# (optional, default: no)
ess_cert_id_chain	= no	# Must the ESS cert id chain be included?
				# (optional, default: no)
ess_cert_id_alg		= sha1	# algorithm to compute certificate
				# identifier (optional, default: sha1)

[xnox]

Description: Allow either Fetcher or OpenSSL exceptions when using invalid cert in rubygems testcase.
Author: Dimitri John Ledkov <xnox@ubuntu.com>
Bug-Upstream: https://github.com/rubygems/rubygems/issues/2388

--- ruby2.5-2.5.1.orig/test/rubygems/test_gem_remote_fetcher.rb
+++ ruby2.5-2.5.1/test/rubygems/test_gem_remote_fetcher.rb
@@ -842,7 +842,7 @@ PeIQQkFng2VVot/WAQbv3ePqWq07g1BBcwIBAg==
     with_configured_fetcher(
       ":ssl_ca_cert: #{temp_ca_cert}\n" +
       ":ssl_client_cert: #{temp_client_cert}\n") do |fetcher|
-        assert_raises Gem::RemoteFetcher::FetchError do
+        assert_raises Gem::RemoteFetcher::FetchError, OpenSSL::SSL::SSLError do
           fetcher.fetch_path("https://localhost:#{ssl_server.config[:Port]}/yaml")
         end
     end

Is what I had to do on Ubuntu. I'm compiling openssl 1.1.1 with ruby2.5.1 with backported ruby-openssl 2.1.1 with TLS security level lowered to 0 in openssl. This seems to work when compiling against both openssl 1.1.0 and 1.1.1.

I suspect that maybe the invalid client cert, is not good enough for tls 1.3 / openssl 1.1.1? or the authentication fails async, thus openssl exception is raised first, or something like that.

[MSP-Greg]

@voxik

OpenSSL 1.1.1 is now the standard MinGW package on Appveyor, and Ruby 2.5.3 and ruby-loco (trunk) are both build with 1.1.1. The above test recently failed on a 2.5.3 job.

But, given that the same test is ran with every ruby/ruby build, every ruby-loco build, and every RubyGems build, it is an intermittent issue.

I added a message to the issue mentioned above...

[MSP-Greg]

@hsbt

Sorry for the ping, but this involves ruby/ruby, RubyGems, & ruby/openssl. I've had this error

TestGemRemoteFetcher#test_do_not_allow_invalid_client_cert_auth_connection

in my RG fork, and I just had it in ruby-loco test-all. It is intermittent and only appears when using OpenSSL 1.1.1 (TLSv1_3).

From what I can tell, the error is originating in one of the OpenSSL dll's, passing thru ruby/openssl, ruby/net/http, ending up raised in RG.

I believe the method Gem::Request#perform_request (which has several rescue statements) would need to have a rescue for OpenSSL::SSL::SSLError. Several of the rescue's output via verbose, I'm not sure whether to use that or say, etc.

As mentioned above, there is an open issue in ruby/openssl #227.

I believe with normal Appveyor & Travis CI, the only way to use OpenSSL 1.1.1 is via Appveyor Ruby 2.5.3 or ruby-loco. Travis Ubuntu is using 1.0.1 (why?), and Travis OSX uses 1.1.0. Ruby trunk MinGW is built/tested with 1.1.1 (mswin is still using 1.0.2, although 1.1.1 is available).

JFYI, from work in another repo that compiles with OpenSSL, the order of the handshakes cb's changes between TLSv1_2 and TLSv1_3. Also, in the above test, I believe the call that generates the error in TLSv1_2 is from fetch_path, but TLSv1_3 errors from perform_request...

After all that, if I did a PR, should I use say or verbose, or would you like to do it?

[voxik]

This is likely resolved by #2507