Ruby's Travis is not working with OpenSSL 1.1.1

[k0kubun]

Recently we added osx build that installs OpenSSL 1.1.1 to Travis of ruby repository. As we got 2 test failures by that, tentatively we skipped those test cases on Travis osx build ruby/ruby@40d0708.

• Deterministic failure: OpenSSL::SSL::SSLError "SSL_read: tlsv1 alert decrypt error"
  • https://travis-ci.org/ruby/ruby/jobs/444240249
• Random failure: Errno::EPROTOTYPE "Protocol wrong type for socket"
  • https://travis-ci.org/ruby/ruby/jobs/444232675

This issue failure seems related to OpenSSL. Once that's fixed, please remove the skip code for Travis.

[k0kubun]

The way of skip was slightly changed ruby/ruby@bf26f84

[rhenium]

Thank you for taking a look at these issues.

• Deterministic failure: OpenSSL::SSL::SSLError "SSL_read: tlsv1 alert decrypt error"

TLS 1.3, which is new in OpenSSL 1.1.1, handles a client certificate differently from TLS 1.2. rubygems/rubygems#2388 is tracking the issue.

• Random failure: Errno::EPROTOTYPE "Protocol wrong type for socket"

I don't know about macOS, but https://bugs.ruby-lang.org/issues/14713 seems relevant. I guess we should apply a patch like this? /cc @nobu

diff --git a/ext/openssl/ossl_ssl.c b/ext/openssl/ossl_ssl.c
index c6bfb8312d91..be596ed19fae 100644
--- a/ext/openssl/ossl_ssl.c
+++ b/ext/openssl/ossl_ssl.c
@@ -1820,6 +1820,14 @@ ossl_ssl_write_internal(VALUE self, VALUE str, VALUE opts)
                 rb_io_wait_readable(FPTR_TO_FD(fptr));
                 continue;
 	    case SSL_ERROR_SYSCALL:
+#ifdef __APPLE__
+		/*
+		 * send can return EPROTOTYPE if the socket is torn down in the
+		 * middle of the syscall. Retry to get a proper errno.
+		 */
+		if (errno == EPROTOTYPE)
+		    continue;
+#endif
 		if (errno) rb_sys_fail(0);
 	    default:
 		ossl_raise(eSSLError, "SSL_write");

[MSP-Greg]

Windows (RubyInstaller2) Ruby 2.5.3 is using 1.1.1, and also ruby-loco trunk.

The failure OpenSSL::SSL::SSLError "SSL_read: tlsv1 alert decrypt error recently occurred in a RubyGems Appveyor build using 2.5.3, so it would seem that this isn't platform specific. The failed RubyGems test expects an error to be raised, but OpenSSL::SSL::SSLError isn't being accounted for.

Also, this is intermittent:

1. Most RubyGems Appveyor CI's have been passing on 2.5.3 & trunk.
2. The Appveyor MinGW OpenSSL package was updated to 1.1.1 no later than 2018-11-10 10:00:29 UTC, and AFAIK, the Ruby Core MinGW build has been passing, or at least passing the RubyGems test.
3. Ruby-loco has been using 1.1.1 for quite a bit longer, and also has been stable (the RubyGems test is part of test-all, but you knew that).

I have not found a way to repro the error locally. The normal error raised (using 1.1.1) is:

Errno::ECONNRESET
An existing connection was forcibly closed by the remote host.

Happy to help, but given that this is moving from RubyGems -> net/http -> openssl, I'm not sure what, where, or who. The RubyGems code could just allow for the error...