jgem doesn't work - handshake_failure talking to rubygems.org

[lpar]

Unable to download data from https://rubygems.org/ - Received fatal alert: handshake_failure (https://rubygems.org/latest_specs.4.8.gz)

JRE is IBM Java(TM) SE Runtime Environment (build pxa6470_27sr1fp1-20140708_01(SR1 FP1))
IBM J9 VM (build 2.7, JRE 1.7.0 Linux amd64-64 Compressed References 20140707_205525 (JIT enabled, AOT enabled)

[lpar]

Sample Java program which works:

import java.io.BufferedInputStream;
import java.io.BufferedOutputStream;
import java.net.URL;
import java.nio.file.FileSystems;
import java.nio.file.Files;
import java.nio.file.Path;

public class TrivialHttpsTest {

  public static void main(final String[] args) {
    try {
      final URL rubygems = new URL("https://rubygems.org/latest_specs.4.8.gz");
      try (BufferedInputStream in = new BufferedInputStream(rubygems.openStream())) {
        final Path latestSpecs = FileSystems.getDefault().getPath("latest_specs.4.8.gz");
        try (BufferedOutputStream out = new BufferedOutputStream(Files.newOutputStream(latestSpecs))) {
          long bytes = 0;
          int b;
          while ((b = in.read()) != -1) {
            bytes += 1;
            out.write(b);
          }
          System.out.println("\nRead " + Long.toString(bytes) + " bytes");
        }
      }
    } catch (final Exception e) {
      e.printStackTrace();
    }
  }

}

...so it's something to do with JRuby or the Ruby stdlib.

[lpar]

Seems to be specific to the IBM JRE. On Oracle JVM 1.8.0_25 it works, same JRuby (latest).

Anything I can do to help track down the cause?

[lpar]

OK, I think I've found a fix. I tried a quick minimal HTTPS fetch script. Turns out, you need to set the OpenSSL ssl_version variable to :TLSv1 and then it'll connect. Without that, it tries to connect using SSLv3, and fails. Since RubyGems.org is now TLS only, this should probably be incorporated into the jgem command.

#!/usr/bin/env ruby
# encoding: UTF-8

require "net/http"

uri = URI.parse("https://rubygems.org/latest_specs.4.8.gz")
http = Net::HTTP.new(uri.host, uri.port)
http.use_ssl = true
http.verify_mode = OpenSSL::SSL::VERIFY_NONE
http.ssl_version = :TLSv1                     #### This line makes it work
request = Net::HTTP::Get.new(uri.request_uri)
response = http.request(request)
puts response.body

Without the ssl_version set, you get the following error:

OpenSSL::SSL::SSLError: Received fatal alert: handshake_failure
   connect at org/jruby/ext/openssl/SSLSocket.java:180
   connect at /usr/local/jruby-1.7.16.1/lib/ruby/1.9/net/http.rb:799
   timeout at org/jruby/ext/timeout/Timeout.java:104
   connect at /usr/local/jruby-1.7.16.1/lib/ruby/1.9/net/http.rb:799
  do_start at /usr/local/jruby-1.7.16.1/lib/ruby/1.9/net/http.rb:755
     start at /usr/local/jruby-1.7.16.1/lib/ruby/1.9/net/http.rb:744
   request at /usr/local/jruby-1.7.16.1/lib/ruby/1.9/net/http.rb:1292
    (root) at testhttp.rb:12

[headius]

Interesting. If this isn't a configuration difference within openssl (that we should make to jruby-openssl) then I guess it would make sense to set this explicitly. I will investigate.

[lpar]

It looks to be something within Ruby's net/http which in turn causes OpenSSL options to be set. Also, the flag isn't a complete fix -- even with that flag, JRuby wouldn't connect to an Apache server I needed to connect to. Unfortunately it's an internal Intranet site, so I can't easily come up with a demonstration right now, but it's Apache on CentOS 6 with a 2048 bit RSA key and SHA2-256 certificate and SSLv3 disabled, I guess when I get some time I could try setting up a VM with similar config and pointing JRuby at that and seeing if it fails.

[headius]

If you are able to set up a VM we can use to reproduce, that would be an enormous help.

[leckylao]

thanks @lpar , I had the exact same issue and http.ssl_version = :TLSv1 helps. 👍

[docwhat]

@headius To reproduce (reasonably reliably for us) just grab "ibm-sdk-7.0.5" and JRuby 1.7.18 on RHEL 6.6 and try gem install bundler --conservative.

If you really need a VM image (like a virtualbox image or something) I think we can probably make one for you. Otherwise I'm not sure what you mean by "set up a VM we can use to reproduce".

[bruceadams]

The latest IBM Java fixes avoid this issue. For more info, see: http://www-01.ibm.com/support/docview.wss?uid=swg21688165

[bruceadams]

To be unambiguous, here is the failure:

$ ruby -v
jruby 1.7.18 (1.9.3p551) 2014-12-22 625381c on IBM J9 VM pxa6470sr7fp1-20140708_01 (SR7 FP1) +jit [linux-amd64]
$ gem install bundler --conservative
ERROR:  Could not find a valid gem 'bundler' (>= 0), here is why:
          Unable to download data from https://rubygems.org/ - Received fatal alert: handshake_failure (https://rubygems.org/latest_specs.4.8.gz)

And here, my same Ubuntu 14.04 laptop, is success:

$ ruby -v
jruby 1.7.18 (1.9.3p551) 2014-12-22 625381c on IBM J9 VM pxa6470_27sr2-20141026_01 (SR2) +jit [linux-amd64]
$ gem install bundler --conservative
Fetching: bundler-1.7.11.gem (100%)
Successfully installed bundler-1.7.11
1 gem installed

[docwhat]

Thanks for figuring that out @bruceadams; Only applies to not-most-recent IBM Javas.

On the one hand, I think it's great all these sites are upgrading their encryption to require TLS (hopefully v1.2+).

On the other hand, we keep having to deal with this.

@headius -- I think it is safe to say that rubygems.org will require TLS for the foreseeable future. I'd suggest just make jgem require TLS out-of-the box. Then this will work with older (but still current-ish) JREs.

[docwhat]

@bruceadams pointed me at this article on IBM's site: http://www-01.ibm.com/support/docview.wss?uid=swg21688165 which probably explains what is going on here.

[headius]

Our jgem command is basically vanilla RubyGems "gem" command and downstream...so there's a missing detail here. If MRI is defaulting to TLSv1 that's cool, we can do the same. If they're not...we need to figure out why we see issues defaulting to SSLv3.

This is very likely related to the "bad_record_mac" errors people used to see...I'm guessing TLSv1 deals with renegotiation better, or something?

Can one of you investigate what the default ssl_version is on MRI...and why?

[lpar]

They changed the behavior for MRI + OpenSSL recently:
https://www.ruby-lang.org/en/news/2014/10/27/changing-default-settings-of-ext-openssl/
https://bugs.ruby-lang.org/issues/9424
https://gist.github.com/jmhodges/d6480f5f81f25b0dfa15

It would probably be a good idea to make JRuby enable TLS 1.1 and 1.2 and disable SSLv3 and TLS 1.0 by default via:

-Ddeployment.security.SSLv2Hello=false -Ddeployment.security.SSLv3=false 
-Ddeployment.security.TLSv1=false -Ddeployment.security.TLSv1.1=true 
-Ddeployment.security.TLSv1.2=true

for people still running Oracle Java 7. And for IBM JDK/JRE,

-Dcom.ibm.jsse2.disableSSLv3=true -Dhttps.protocols="TLSv1.1,TLSv1.2"

should do the trick. (Edit: I think that should be disableSSLv3 = true, not false, because the sense of the option is reversed.)

[headius]

@lpar Awesome, thank you. This is actionable and I think it can happen for 1.7.19.

@enebo Yes?

[headius]

Copying @kares, @mkristian since they have been maintaining jruby-openssl recently.

[lpar]

Note that you can also make the property changes via Java code if it's more convenient, so long as you do so before opening the first https connection. (That's how I ended up fixing my applications which are plain runnable .jar files.)

[docwhat]

As near as I can tell, IBMs Java doesn't support renegotiation by default at all. OpenSSL by default supports renegotiation upwards (e.g. SSLv3 to TLS). I think Oracle Java is doing the same.

One of the post POODLE bugs was downwards renegotiation, which can allow SSLv3 attacks when initially requesting TLS. There was something similar with 3 -> 2. So maybe IBM was being overly cautious and disabled all renegotiation as default.

I like being explicit. It's too bad we'll have keep adjusting as new TLS versions come out; 1.3 is due soon.

[bruceadams]

I stood up a little VM on DigitalOcean with the IP address 104.236.104.183. It has four IBM Java versions installed:

/opt/ibm-java/sdk-6.0-16.1
/opt/ibm-java/sdk-6.0-16.2
/opt/ibm-java/sdk-7.1-1.1
/opt/ibm-java/sdk-7.1-2.0

and JRuby itself installed:

/opt/jruby-1.7.18

There is a user login for @headius with his github public ssh keys. I'm happy to setup a user login for anyone else who wants a test machine with IBM Java already installed.

On that machine, as a non-privileged user:

ba@j9:~$ JAVA_HOME=/opt/ibm-java/sdk-7.1-1.1 gem install -i 7bad bundler --conservative
ERROR:  Could not find a valid gem 'bundler' (>= 0), here is why:
          Unable to download data from https://rubygems.org/ - Received fatal alert: handshake_failure (https://rubygems.org/latest_specs.4.8.gz)
ba@j9:~$ JAVA_HOME=/opt/ibm-java/sdk-7.1-2.0 gem install -i 7good bundler --conservative
Fetching: bundler-1.7.11.gem (100%)
Successfully installed bundler-1.7.11
1 gem installed

[docwhat]

@bruceadams 👌

[FilBot3]

I also seem to be having this issue as well on RHEL 5.8 using IBM JVM9 1.6 SR2

My IBM JVM information

~]# /opt/IBM/WebSphere/AppServer/java/bin/java -version
java version "1.6.0"
Java(TM) SE Runtime Environment (build pxa6460_26sr2fp1-20120622_01(SR2 FP1))
IBM J9 VM (build 2.6, JRE 1.6.0 Linux amd64-64 20120618_113791 (JIT enabled, AOT enabled)
J9VM - R26_Java626_SR2_FP1_20120618_1531_B113791
JIT  - r11_20120322_22976ifx3
GC   - R26_Java626_SR2_FP1_20120618_1531_B113791
J9CL - 20120618_113791)
JCL  - 20120621_01

The script I'm using to run JRuby-Complete JAR

~]# cat run_jruby.sh
#!/bin/bash

JAVA_COMMAND='/opt/IBM/WebSphere/AppServer/java/bin/java'
JAVA_OPTIONS='-jar -Dcom.ibm.jsse2.disableSSLv3=true -Dhttps.protocols="TLSv1.1,TLSv1.2"'
JRUBY_BIN='/opt/IBM/jruby/jruby-complete-1.7.21.jar'

export GEM_HOME='/opt/IBM/jruby/vendor/gem_home'
export GEM_PATH='/opt/IBM/jruby/vendor/gem_home'

${JAVA_COMMAND} ${JAVA_OPTIONS} ${JRUBY_BIN} $*

The JRuby Version Information

~]# ./run_jruby.sh -v
jruby 1.7.21 (1.9.3p551) 2015-07-07 a741a82 on IBM J9 VM pxa6460_26sr2fp1-20120622_01 (SR2 FP1) +jit [linux-amd64]

The RHEL OS Information

~]# uname -a
Linux server00 2.6.18-308.24.1.el5 #1 SMP Wed Nov 21 11:42:14 EST 2012 x86_64 x86_64 x86_64 GNU/Linux

and the error output from trying to download the Bundler RubyGem

~]# ./run_jruby.sh -S gem install -i 'vendor/gem_home' --no-ri --no-rdoc bundler --verbose
ERROR:  Could not find a valid gem 'bundler' (>= 0), here is why:
          Unable to download data from https://rubygems.org/ - Received fatal alert: handshake_failure (https://api.rubygems.org/specs.4.8.gz)

Hopefully this helps a bit. My other issues with JRuby not running Rake was just on Windows so far.

[bruceadams]

That is a very old Java release, from June 2012. Any chance you can upgrade Java to a recent release? There have been several SSL/TLS related problems and fixes over the past year, starting with POODLE.

[FilBot3]

@bruceadams unfortunately no, due to lack of budget from my place of business (grumble grumble).

[kares]

turns out there's been SSL cert matching issues in jruby-openssl, also tracked at #4679