[2.1.10] gem update fail on windows

[jonforums]

Using 32bit MRI on 64bit Win8.1 I get a bogus failure when attempting to update the sequel gem. The same failure does not occur on a 64bit MRI on 64bit Ubuntu Server 13.10.

I don't use ruby much these days, but here's an initial debug. Given the --debug output it may be MRI not RG.

C:\>ruby -ropenssl -v -e "puts OpenSSL::OPENSSL_VERSION"
ruby 2.0.0p343 (2013-10-31 revision 43481) [i386-mingw32]
OpenSSL 1.0.1e 11 Feb 2013

C:\>gem --version
2.1.10

C:\>gem out
sequel (4.3.0 < 4.4.0)

C:\>gem up sequel --verbose
Updating installed gems
HEAD https://rubygems.org/specs.4.8.gz
302 Moved Temporarily
HEAD https://s3.amazonaws.com/production.s3.rubygems.org/specs.4.8.gz
304 Not Modified
Updating sequel
GET https://rubygems.org/quick/Marshal.4.8/sequel-4.4.0.gemspec.rz
302 Moved Temporarily
ERROR:  While executing gem ... (Gem::SpecificGemNotFoundException)
    Could not find a valid gem 'sequel' (= 4.4.0) locally or in a repository

C:\>gem up sequel --debug
Exception `NoMethodError' at C:/ruby200-x32/lib/ruby/site_ruby/2.0.0/rubygems/specification.rb:1890 - undefined method `to_ary' for #<Gem::Specification:0x32e4628>
...[SNIP]...
Updating installed gems
Exception `Errno::EEXIST' at C:/ruby200-x32/lib/ruby/2.0.0/fileutils.rb:245 - File exists - C:/Users/Jon/.gem/specs/rubygems.org%443
Exception `OpenSSL::SSL::SSLError' at C:/ruby200-x32/lib/ruby/2.0.0/openssl/buffering.rb:174 - read would block
Exception `OpenSSL::SSL::SSLError' at C:/ruby200-x32/lib/ruby/2.0.0/openssl/buffering.rb:174 - read would block
Updating sequel
Exception `OpenSSL::SSL::SSLError' at C:/ruby200-x32/lib/ruby/2.0.0/openssl/buffering.rb:174 - read would block
Exception `OpenSSL::SSL::SSLError' at C:/ruby200-x32/lib/ruby/2.0.0/net/http.rb:918 - SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed
Exception `OpenSSL::SSL::SSLError' at C:/ruby200-x32/lib/ruby/2.0.0/net/http.rb:926 - SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed
Exception `Gem::RemoteFetcher::FetchError' at C:/ruby200-x32/lib/ruby/site_ruby/2.0.0/rubygems/request.rb:102 - SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed (https://rubygems.global.ssl.fastly.net/quick/Marshal.4.8/sequel-4.4.0.gemspec.rz)
Exception `Gem::RemoteFetcher::FetchError' at C:/ruby200-x32/lib/ruby/site_ruby/2.0.0/rubygems/remote_fetcher.rb:271 - SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed (https://rubygems.global.ssl.fastly.net/quick/Marshal.4.8/sequel-4.4.0.gemspec.rz)
Exception `Gem::SpecificGemNotFoundException' at C:/ruby200-x32/lib/ruby/site_ruby/2.0.0/rubygems/dependency_installer.rb:271 - Could not find a valid gem 'sequel' (= 4.4.0) locally or in
a repository
ERROR:  While executing gem ... (Gem::SpecificGemNotFoundException)
    Could not find a valid gem 'sequel' (= 4.4.0) locally or in a repository
        C:/ruby200-x32/lib/ruby/site_ruby/2.0.0/rubygems/dependency_installer.rb:271:in `find_spec_by_name_and_version'
...[SNIP]...

[luislavena]

@jonforums the issue seems to be truncated. Can you share the output?

Thank you.

[jonforums]

This looks to be another cert issue. When I add the cert from

http://curl.haxx.se/docs/caextract.html

to C:\ruby200-x32\lib\ruby\site_ruby\2.0.0\rubygems\ssl_certs, the above scenario succeeds.

Perhaps https://rubygems.global.ssl.fastly.net/ is not covered by the new RG certs?

[jonforums]

FWIW, the above scenario also succeeds when I place a DigiCertHighAssuranceEVRootCA.pem created by

C:\devlibs\openssl\x86>openssl.exe x509 -in DigiCertHighAssuranceEVRootCA.crt -out DigiCertHighAssuranceEVRootCA.pem -outform PEM

into site_ruby\2.0.0\rubygems\ssl_certs.

I obtained the root CA crt by browsing to https://a.ssl.fastly.net/ and downloading the certs when firefox issued an untrusted connection warning.

[olivierdagenais]

When I add the cert from

http://curl.haxx.se/docs/caextract.html

to C:\ruby200-x32\lib\ruby\site_ruby\2.0.0\rubygems\ssl_certs, the above scenario succeeds.

I also fixed my Ruby installation on Windows by doing this.

[jonforums]

@olivierdagenais it appears @drbrain already knew about and fixed the issue with 0b5185e a few days before I posted.

The new pem just hasn't been rolled out in a new RG update yet. That said, you can simply download the raw file and stick it in your ssl_certs dir and things should be fine.

Remove the curl certs bundle and try the new pem to see if it also works for you.

[tmilker]

Is this issue(in general) ever going to be permanently fixed? I've run into this multiple times over the last couple months and it is extremely annoying. It really screws up automatic provisioning. I also wish gem sources --add had a force or --yes-i-know-what-i-am-doing flag to skip the [yn] check for adding an http source(the temporary workaround I use while this is happening).

[drbrain]

@tmilker this issue can never be permanently fixed by the RubyGems software. If you provide an up-to-date certificate set when you provision software you should have no problems.

I have a script here which I use to check for new certs when things get out of date. Unfortunately due to RubyConf I haven't been able to find the time to release the change, but I should have time today.

Not that this issue can never be fixed permanently because our hosting infrastructure may change their SSL certificates without notice. This is where providing an up-to-date certificate set will avoid these problems. RubyGems uses both its built-in certificates and the system certificates when validating SSL connections.

Also note that we may need to alter the infrastructure used to host rubygems.org due to changes in bandwidth cost, errors or other reasons. This may break older versions of rubygems (the client software). Having an up-to-date certificate set will avoid these problems.

[tmilker]

Thanks for the script, I'll see what I can do about integrating it with what I'm doing.

Doesn't this pain point indicate that the certs are being distributed/stored incorrectly then?

This is just off the top of my head, I really have no idea how rubygems does all it's magic(though I appreciate it a lot, even when SSL certs are driving me crazy) so this may not even be feasible:

Why not allow rubygems users to tell rubygems to update its certs, independent of the rubygems version? If the infrastructure itself is versioned, then rubygems will know if it's out-of-date and can check that separately. Breaking the dependency between the rubygems version and the current set of certs, which have nothing to do with each other when the infrastructure is the same, seems like a good thing.

[jonforums]

@tmilker there may be a circular problem with automated cert updates. For example, RG infrastructure changes and the cert to RG (containing its updated certs) becomes outdated. How should RG do a secure cert update without opening itself up to downloading a potentially malicious cert?

I don't know the answer to this and haven't looked into RG code in a long time. Perhaps @drbrain may have time to reply whether this is a legitimate scenario.

That said, automagically updating certs and the corner cases concerns me.

I'd be OK with a quick fix to RG that makes the issue more visible. Perhaps a new Exception type and messaging that get's around the misleading Gem::SpecificGemNotFoundException that messages Could not find a valid gem...

[drbrain]

We currently allow rubygems users to update certs through openssl.

While the infrastructure is versioned, the SSL certificates used by our CDNs is not, and we don't get advance notice of changes there. Providing a way to update them through rubygems seems perilous as the curl list is served over HTTP (not HTTPS). See also @jonforums' for the other problem with this.

We could instead ship a complete set of trusted certificates like browsers do, but this adds a lot of maintenance burden. If one of the certificates is compromised we need to quickly act to remove the certificate from the rubygems client through an update even when we don't use it. We need to act quickly when certificate vendors do untrustworthy things. Browser vendors have a staff on hand to watch this, but we don't have the resources to allow this, so the risk is too high.

[drbrain]

I should add that the OpenSSL extension doesn't give good feedback on why there was a certificate validation failure. The following can be true:

• connection was disconnected
• client has is missing the CA certificate
• server has a forged certificate
• MITM

There may be others, but if we could distinguish between disconnect, missing CA cert and other validation failures we could give much better messaging.

[jonforums]

Does it add any value to have a generic certificate validation failure exception?

While not specific, at least it would alert users that this issue is a cert issue (or potentially bad network connection) rather than pointing them in a wrong direction. They'll need to go debug, but they'll be pointed in the correction direction.

[drbrain]

I think there's value to that, but I haven't had time to add such a thing with Ruby 2.1 prep work ☹

[jonforums]

If it needs a foundational tweak to the OpenSSL extension to surface more relevant info for use by a new RG exception, would posting it as an OpenSSL issue on ruby-core's redmine help?

I don't spend much time with ruby these days, but I seem to recall one of the other core committers helping you with RG specific code drops. Perhaps you could sweet talk one of them into updating OpenSSL extension and sending you a RG pull request 😸

[drbrain]

@emboss is the openssl maintainer for ruby, so opening an issue in bugs.ruby-lang.org will help, but I'm unsure if openssl gives you better indication than what the extension provides.

Rescuing the exception in RubyGems and raising an exception with a more helpful message would be a stopgap solution, but I haven't had the time to implement that.

[drbrain]

I have released 2.1.11 (and 2.0.14) with the DigiCert certificate, can someone check it?

[jonforums]

LGTM...

C:\>uru ls
 => 200p349-x32 : ruby 2.0.0p349 (2013-11-12 revision 43656) [i386-mingw32]

C:\>gem --version
2.1.10

C:\>gem up --system
Updating rubygems-update
Fetching: rubygems-update-2.1.11.gem (100%)
Successfully installed rubygems-update-2.1.11
Installing RubyGems 2.1.11
RubyGems 2.1.11 installed

...SNIP...

RubyGems installed the following executables:
        C:/ruby200-x32/bin/gem

RubyGems system software updated


C:\>gem --version
2.1.11

C:\>gem out
oj (2.1.7 < 2.2.0)
sequel (4.3.0 < 4.4.0)

C:\>gem up sequel
Updating installed gems
Updating sequel
Fetching: sequel-4.4.0.gem (100%)
Successfully installed sequel-4.4.0
Gems updated: sequel

[drbrain]

Thanks, I will close this issue.

[olivierdagenais]

It seems the migration to a CDN could have been handled by introducing https://cdn.rubygems.org from version 2.1.11 and up (i.e. only switch the sources to the CDN at the same time the new certificate is provided), that way it would have allowed simple updates to still use https://rubygems.org (which could even be throttled down to discourage its continued use).

[drbrain]

@olivierdagenais We've been using a CDN and S3 to serve gems for some time now, so this solution won't work for all cases. It's still possible that S3 or cloudfront can change their certificates at any time, or it may be necessary for us to switch CDNs without notice due to errors or down time.