Latest cacerts breaks SSL to Amazon / AWS

[jefmathiot]

The cacerts software is fetching cURL certificates bundle and makes it available to the embedded OpenSSL. The cURL bundle is converted from mozilla.org using the mk-ca-bundle (Perl) script.

The issue there is that Mozilla marked some certificates for later removal / deprecation, including Verisign's. The certificates are valid, but the script don't add them to the bundle. Amazon's certificates are generated using this CA thus the newly built packages won't be able to verify any of the Amazon & AWS SSL certificates (including EC2 API, S3, etc):

mkdir /tmp/empty-ca-path

# With the latest CA bundle
wget http://curl.haxx.se/ca/cacert.pem -O /tmp/latest-cacert.pem
curl -v --capath /tmp/empty-ca-path --cacert /tmp/latest-cacert.pem \
  https://s3-eu-central-1.amazonaws.com
# => Verification failed

# With the last CA bundle before Mozilla cleanup
wget https://raw.githubusercontent.com/bagder/ca-bundle/e9175fec5d0c4d42de24ed6d84a06d504d5e5a09/ca-bundle.crt -O /tmp/before-cleanup.pem
curl -v --capath /tmp/empty-ca-path --cacert /tmp/before-cleanup.pem \
  https://s3-eu-central-1.amazonaws.com
# => Verification successful

I've encountered the issue on a gem which uses Amazon S3 amongst other services.

It seems that the perl script allows inclusion of these certificates using the -p flag so these are the potential fixes I've identified so far:

• download the latest working bundle
• download the latest bundle and add the missing certificates
• execute the script with the appropriate option instead of downloading the bundle (would require perl on the build machine)
• make a port of the script to Ruby
• ?

I'd be glad to create a pull request but I'm not sure of the strategy to adopt.

[lamont-granquist]

Are they not working to fix that at the curl url? We've just pinned the version to the prior version before those got yanked, and I thought they were going to do what you're suggesting and then update how they use that script. Its not just omnibus-chef that is affected, anyone else that pulls that bundle will break talking to s3 and will complain to them again, so I would expect that they'll have to fix it sooner or later.

[jefmathiot]

@lamont-granquist Thanks for the quick reply. I posted a followup to cURL's mailing list (I didn't realize you were the author of the original post). Until this problem is resolved and following the Principle of Least Astonishment, maybe documenting the issue somewhere could contribute to the mental health of omnibus users ? Let me know if I can be of any help.

[wolf31o2]

Meh... this just bit me, too. I ended up allowing the individual versions to set a source_file variable, which defaults to cacert.pem and specifying the source :url in my version and editing the file copy to use source_file by variable, rather than cacert.pem by string.

[lamont-granquist]

Omnibus already supports caching in s3, our version of the cacerts that we're using is here:

s3://opscode-omnibus-cache/cacerts-2014.07.15-fd48275847fa10a8007008379ee902f1

You can override that bucket in your project and point someplace else and upload that file to your bucket to do your own s3 caching. For production use, you pretty much have to turn on s3 caching because urls to software disappear all the time.

[schisamo]

Here is an example of pinning to the version @lamont-granquist referenced:
https://github.com/chef/omnibus-chef/blob/master/config/projects/chef.rb#L41-L51

[wolf31o2]

Forcing users to use a pay service to get the software to build seems... weak.

Why not follow the docs and use the files from github over HTTPS, which don't change, versus the HTTP website version which does?

Here's the last CA bundle from before the cleanup (2014.08.13):
https://github.com/bagder/ca-bundle/blob/e9175fec5d0c4d42de24ed6d84a06d504d5e5a09/ca-bundle.crt

Here's the latest from after (2015.02.25):
https://raw.githubusercontent.com/bagder/ca-bundle/d82fc46afaf6478aaa22989bcc2202ba7b72ad71/ca-bundle.crt

[schisamo]

@wolf31o2 Thanks for the pointer. Mind opening a PR with fix?

[wolf31o2]

Done... :-)

[jefmathiot]

@wolf31o2 @schisamo Pinning the version using an override in the projects is way cleaner than our approach, but we were not very comfortable with it because the default version of the bundle used (2015.02.25) breaks compatbility with S3, which in my opinion violates the principle of least surprise for omnibus users. For that reason we use the version pointed by @wolf31o2 (extracted from github) in a fork of omnibus-software https://github.com/servebox/omnibus-software/blob/pin-cacerts/config/software/cacerts.rb

I'm not sure of the potential side effects, but maybe it could make sense to use the last working version as the default?

[wolf31o2]

That's an easy change to the PR in #361 which uses the new files.

[lamont-granquist]

fixed by #554 (plus i think Amazon finally got off the root cert that was using MD5)