SSL "Certificate verify failed" when running trivial test program

[andy-twosticks]

Here's my trivial test program:

require 'rss'
require 'open-uri'

def pull(url)
  open(url) do |rss|
    feed = RSS::Parser.parse(rss)
    feed.items.each{ |item| puts item.title }
  end
end

pull 'https://www.theguardian.com/uk-news/rss'

This program works when I run it using ruby. But the compiled (gcc) version does this:

/__enclose_io_memfs__/lib/ruby/2.4.0/net/protocol.rb:44:in `connect_nonblock': SSL_connect returned=1 errno=0 state=error: certificate verify failed (OpenSSL::SSL::SSLError)
        from /__enclose_io_memfs__/lib/ruby/2.4.0/net/protocol.rb:44:in `ssl_socket_connect'
        from /__enclose_io_memfs__/lib/ruby/2.4.0/net/http.rb:948:in `connect'
        from /__enclose_io_memfs__/lib/ruby/2.4.0/net/http.rb:887:in `do_start'
        from /__enclose_io_memfs__/lib/ruby/2.4.0/net/http.rb:876:in `start'
        from /__enclose_io_memfs__/lib/ruby/2.4.0/open-uri.rb:323:in `open_http'
        from /__enclose_io_memfs__/lib/ruby/2.4.0/open-uri.rb:741:in `buffer_open'
        from /__enclose_io_memfs__/lib/ruby/2.4.0/open-uri.rb:212:in `block in open_loop'
        from /__enclose_io_memfs__/lib/ruby/2.4.0/open-uri.rb:210:in `catch'
        from /__enclose_io_memfs__/lib/ruby/2.4.0/open-uri.rb:210:in `open_loop'
        from /__enclose_io_memfs__/lib/ruby/2.4.0/open-uri.rb:151:in `open_uri'
        from /__enclose_io_memfs__/lib/ruby/2.4.0/open-uri.rb:721:in `open'
        from /__enclose_io_memfs__/lib/ruby/2.4.0/open-uri.rb:35:in `open'
        from /__enclose_io_memfs__/local/one.rb:6:in `pull'
        from /__enclose_io_memfs__/local/one.rb:12:in `<main>'

Not sure what is going on here; I'm far from clever with SSL.

If change 'https://' to 'http://' in the code, then there is no change: it still retrieves the headlines in ruby, and still returns the exact same error message when compiled.

[pmq20]

Thanks for reporting. Let me re-produce it locally and debug ASAP.

[andy-twosticks]

FYI: Under 0.4.0, no change.

[yourivdlans]

Probably the compiled package cannot use the certificates on the system and it doesn't have any included. You could use something like this gem https://github.com/liveeditor/net_http_ssl_fix to fix the issue.

[andy-twosticks]

@yourivdlans
Okay. But it doesn't do this in Ruby. And given that this is a very simple test program that only uses stdlib, shouldn't this be documented as part of the install instructions, at least?

I mean, I would really like the Ruby deployment story to be better. So I tried this with a simple program and it didn't work. So I gave up on it, nearly a year ago. Which is presumably not what you want people to do.

[yourivdlans]

@andy-twosticks I fully agree, its not ideal. Compiled programs not using https will probably run fine. I assume the developers hadn't seen this edge case before.

Last resort could also be to disable the certificate verification, but also not an ideal solution.

[drbrain]

Can you run:

ruby -e "require 'openssl'; puts OpenSSL::X509::DEFAULT_CERT_FILE"

and:

ENCLOSE_IO_ALWAYS_USE_ORIGINAL_RUBY=1 ENCLOSE_IO_USE_ORIGINAL_RUBY=1 ./a.out -e "require 'openssl'; puts OpenSSL::X509::DEFAULT_CERT_FILE"

If these are not the same directory then we need to tweak openssl configuration.

I think you can work around the problem by setting SSL_CERT_FILE to the one ruby shows.

[yourivdlans]

This is my output:

$ ruby -e "require 'openssl'; puts OpenSSL::X509::DEFAULT_CERT_FILE"
/usr/local/etc/openssl/cert.pem
$ ENCLOSE_IO_ALWAYS_USE_ORIGINAL_RUBY=1 ENCLOSE_IO_USE_ORIGINAL_RUBY=1 ./a.out -e "require 'openssl'; puts OpenSSL::X509::DEFAULT_CERT_FILE"
/usr/local/ssl/cert.pem

[drbrain]

I think rubyc can configure openssl to change where DEFAULT_CERT_FILE points

[yourivdlans]

I've done a bit of testing and come to the following conclusions:

1. Using 0.4.0 building openssl with --openssldir=/usr/local/etc/openssl will ensure the build succeeds (so gems from https sources get downloaded). But when moving the packaged file to another system that doesn't have the certification in that location it will not be able to do https requests.
2. Using master it seems I can compile without trouble but I'm not able to run a compiled rails app.

Possible solution:

Would it be an idea to download a certification or copy from the local disk during the build process and placing it in the work dir, then pointing openssl to that dir?

Although then there is the possibility the certification will get outdated and the app will break. Not sure what the best solution is here.

[drbrain]

Setting --openssldir can be exposed as an option for the user running rubyc, and a default of /usr/local/etc/openssl seems OK?

[yourivdlans]

That would be ok for people wanting to compile a package. But then when the package gets distributed and other systems don't have the certifications in /usr/local/etc/openssl the program will not be able to do https requests.

[yourivdlans]

I had a look at traveling-ruby to see how they solved this problem and this is what I found.

1. They also compile their own version of openssl but they set the --openssl-dir to the path they compile openssl in.
2. They have a copy of the ca-bundle.crt in their repository.
3. They compile ruby with --with-openssl-dir pointing to their compiled version of openssl.
4. Then they copy the ca-bundle.crt to the dir they are packaging ruby from and setting the SSL_CERT_FILE env var.

So I assume they package root certificates which allows any system / user to run the package regardless of their own certificates installed on the system.

I'd suggest ruby-packer should do something identical, what do you think?

[drbrain]

If we bundle certificates then when CA certificates expire or are revoked rubyc must be recompiled, then users must fetch rubyc, then recompile their application, then distribute their application.

Users can be vulnerable to connection hijacking even when the site they're connecting to doesn't use certificates signed by that CA (although the CA was revoked, it is still included thus still trusted). It will mean a CVE for rubyc if they're bundled inside rubyc, so I don't want to bundle them inside ruby-packer as the maintenance burden is far too high.

Omnibus can download and install the bundle from curl which removes the need to recompile and distribute rubyc, but not the need to recompile the user's application. This is OK, but leaves the expiration and revocation maintenance burden on the application packager and they may not be aware of it.

I prefer to have the packager manage certificates in an informed way. I think it is OK to allow users to make the choice to bundle certificates inside their executable, but it should come with warnings about and protections for the security risks.

As a first pass I would prefer to make the openssl dir configurable so packagers can instruct users in how to configure their CA certificate list. Other applications they deploy may already be configured to use the same store, so this may be a no-op. (Homebrew on macOS allows easy installation and maintenance of CA certificates and I believe most linux distributions have the same. We don't compile OpenSSL on Windows yet, but eventually we'll need to point to someone's method of adding CA certificates for Windows.)

A follow-up may allow users to place trusted CA certificates in a directory that ruby-packer will package into their executable. This should allow users to balance their vulnerability exposure. RubyGems follows this pattern and bundles only the CA certificates for the servers it connects to to limit vulnerability upon revocation of unrelated CA certificates.

[yourivdlans]

Users can be vulnerable to connection hijacking even when the site they're connecting to doesn't use certificates signed by that CA (although the CA was revoked, it is still included thus still trusted).

Very good point.

As a first pass I would prefer to make the openssl dir configurable so packagers can instruct users in how to configure their CA certificate list.

Alright yeah, thats a good first step.

This should allow users to balance their vulnerability exposure. RubyGems follows this pattern and bundles only the CA certificates for the servers it connects to to limit vulnerability upon revocation of unrelated CA certificates.

Really like this idea!

Thanks for the clear explanation. I'll try and create a pull request for that first step today :)

[plindelauf]

I'm running into this issue as well... any hopes of seeing a solution for this?