SSL Failure on Import the NodeSource GPG key into apt

[jwhitlock]

In the last 24 hours, I began getting a failure on the "Import the NodeSource GPG key into apt" step with Ansible 1.9.2

A full run can be seen in TravisCI:
https://travis-ci.org/mozilla/kuma/jobs/158534929

The full URL seems to have a valid certificate:
https://deb.nodesource.com/gpgkey/nodesource.gpg.key

However, the root URL now redirects to GitHub:
https://deb.nodesource.com

I suspect the problem is that Ansible does certificate validation of the root URL, not the full path, and is detecting a problem with the hostname change.

Here's the output from a verbose run:

TASK: [nodesource.node | Import the NodeSource GPG key into apt] ************** 
<127.0.0.1> ESTABLISH CONNECTION FOR USER: vagrant
<127.0.0.1> REMOTE_MODULE apt_key state=present url=https://deb.nodesource.com/gpgkey/nodesource.gpg.key
<127.0.0.1> EXEC ssh -C -tt -vvv -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ForwardAgent=yes -o ControlMaster=auto -o ControlPersist=60s -o ControlPath="/Users/john/.ansible/cp/ansible-ssh-%h-%p-%r" -o StrictHostKeyChecking=no -o Port=2222 -o IdentityFile="/Users/john/src/kuma/.vagrant/machines/developer-local/virtualbox/private_key" -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=vagrant -o ConnectTimeout=30 127.0.0.1 /bin/sh -c 'mkdir -p $HOME/.ansible/tmp/ansible-tmp-1473372799.77-250761793755640 && chmod a+rx $HOME/.ansible/tmp/ansible-tmp-1473372799.77-250761793755640 && echo $HOME/.ansible/tmp/ansible-tmp-1473372799.77-250761793755640'
<127.0.0.1> PUT /var/folders/61/s6_xxhqd3nl27_vgq9fzjmkr0000gq/T/tmpHnT09K TO /home/vagrant/.ansible/tmp/ansible-tmp-1473372799.77-250761793755640/apt_key
<127.0.0.1> EXEC ssh -C -tt -vvv -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ForwardAgent=yes -o ControlMaster=auto -o ControlPersist=60s -o ControlPath="/Users/john/.ansible/cp/ansible-ssh-%h-%p-%r" -o StrictHostKeyChecking=no -o Port=2222 -o IdentityFile="/Users/john/src/kuma/.vagrant/machines/developer-local/virtualbox/private_key" -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=vagrant -o ConnectTimeout=30 127.0.0.1 /bin/sh -c 'sudo -k && sudo -H -S -p "[sudo via ansible, key=peqdhgbtpdvxszbvlonushpnfpjjmcyq] password: " -u root /bin/sh -c '"'"'echo BECOME-SUCCESS-peqdhgbtpdvxszbvlonushpnfpjjmcyq; LANG=en_US.UTF-8 LC_CTYPE=en_US.UTF-8 PYTHONDONTWRITEBYTECODE=1 /usr/bin/python /home/vagrant/.ansible/tmp/ansible-tmp-1473372799.77-250761793755640/apt_key'"'"''
failed: [developer-local] => {"failed": true}
msg: Failed to validate the SSL certificate for deb.nodesource.com:443. Use validate_certs=False (insecure) or make sure your managed systems have a valid CA certificate installed. Paths checked for this platform: /etc/ssl/certs, /etc/pki/ca-trust/extracted/pem, /etc/pki/tls/certs, /usr/share/ca-certificates/cacert.org, /etc/ansible

FATAL: all hosts have already failed -- aborting
TASK: [nodesource.node | Import the NodeSource GPG key into apt] ************** 
<127.0.0.1> ESTABLISH CONNECTION FOR USER: vagrant
<127.0.0.1> REMOTE_MODULE apt_key state=present url=https://deb.nodesource.com/gpgkey/nodesource.gpg.key
<127.0.0.1> EXEC ssh -C -tt -vvv -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ForwardAgent=yes -o ControlMaster=auto -o ControlPersist=60s -o ControlPath="/Users/john/.ansible/cp/ansible-ssh-%h-%p-%r" -o StrictHostKeyChecking=no -o Port=2222 -o IdentityFile="/Users/john/src/kuma/.vagrant/machines/developer-local/virtualbox/private_key" -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=vagrant -o ConnectTimeout=30 127.0.0.1 /bin/sh -c 'mkdir -p $HOME/.ansible/tmp/ansible-tmp-1473372799.77-250761793755640 && chmod a+rx $HOME/.ansible/tmp/ansible-tmp-1473372799.77-250761793755640 && echo $HOME/.ansible/tmp/ansible-tmp-1473372799.77-250761793755640'
<127.0.0.1> PUT /var/folders/61/s6_xxhqd3nl27_vgq9fzjmkr0000gq/T/tmpHnT09K TO /home/vagrant/.ansible/tmp/ansible-tmp-1473372799.77-250761793755640/apt_key
<127.0.0.1> EXEC ssh -C -tt -vvv -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ForwardAgent=yes -o ControlMaster=auto -o ControlPersist=60s -o ControlPath="/Users/john/.ansible/cp/ansible-ssh-%h-%p-%r" -o StrictHostKeyChecking=no -o Port=2222 -o IdentityFile="/Users/john/src/kuma/.vagrant/machines/developer-local/virtualbox/private_key" -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=vagrant -o ConnectTimeout=30 127.0.0.1 /bin/sh -c 'sudo -k && sudo -H -S -p "[sudo via ansible, key=peqdhgbtpdvxszbvlonushpnfpjjmcyq] password: " -u root /bin/sh -c '"'"'echo BECOME-SUCCESS-peqdhgbtpdvxszbvlonushpnfpjjmcyq; LANG=en_US.UTF-8 LC_CTYPE=en_US.UTF-8 PYTHONDONTWRITEBYTECODE=1 /usr/bin/python /home/vagrant/.ansible/tmp/ansible-tmp-1473372799.77-250761793755640/apt_key'"'"''
failed: [developer-local] => {"failed": true}
msg: Failed to validate the SSL certificate for deb.nodesource.com:443. Use validate_certs=False (insecure) or make sure your managed systems have a valid CA certificate installed. Paths checked for this platform: /etc/ssl/certs, /etc/pki/ca-trust/extracted/pem, /etc/pki/tls/certs, /usr/share/ca-certificates/cacert.org, /etc/ansible

FATAL: all hosts have already failed -- aborting

[jwhitlock]

My host machine is OS X running Python 2.7.11, and the managed system is Ubuntu 14.04.4 running Python 2.7.6.

With Ansible 1.9.6, the error is:

TASK: [nodesource.node | Import the NodeSource GPG key into apt] ************** 
<127.0.0.1> ESTABLISH CONNECTION FOR USER: vagrant
<127.0.0.1> REMOTE_MODULE apt_key state=present url=https://deb.nodesource.com/gpgkey/nodesource.gpg.key
<127.0.0.1> EXEC ssh -C -tt -vvv -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ForwardAgent=yes -o ControlMaster=auto -o ControlPersist=60s -o ControlPath="/Users/john/.ansible/cp/ansible-ssh-%h-%p-%r" -o StrictHostKeyChecking=no -o Port=2222 -o IdentityFile="/Users/john/src/kuma/.vagrant/machines/developer-local/virtualbox/private_key" -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=vagrant -o ConnectTimeout=30 127.0.0.1 /bin/sh -c 'mkdir -p $HOME/.ansible/tmp/ansible-tmp-1473374305.67-265645841307425 && chmod a+rx $HOME/.ansible/tmp/ansible-tmp-1473374305.67-265645841307425 && echo $HOME/.ansible/tmp/ansible-tmp-1473374305.67-265645841307425'
<127.0.0.1> PUT /var/folders/61/s6_xxhqd3nl27_vgq9fzjmkr0000gq/T/tmpavvBEM TO /home/vagrant/.ansible/tmp/ansible-tmp-1473374305.67-265645841307425/apt_key
<127.0.0.1> EXEC ssh -C -tt -vvv -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ForwardAgent=yes -o ControlMaster=auto -o ControlPersist=60s -o ControlPath="/Users/john/.ansible/cp/ansible-ssh-%h-%p-%r" -o StrictHostKeyChecking=no -o Port=2222 -o IdentityFile="/Users/john/src/kuma/.vagrant/machines/developer-local/virtualbox/private_key" -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=vagrant -o ConnectTimeout=30 127.0.0.1 /bin/sh -c 'sudo -k && sudo -H -S -p "[sudo via ansible, key=npkujndnrlchfeeibmjxrumgonvoefqf] password: " -u root /bin/sh -c '"'"'echo BECOME-SUCCESS-npkujndnrlchfeeibmjxrumgonvoefqf; LANG=en_US.UTF-8 LC_CTYPE=en_US.UTF-8 PYTHONDONTWRITEBYTECODE=1 /usr/bin/python /home/vagrant/.ansible/tmp/ansible-tmp-1473374305.67-265645841307425/apt_key'"'"''
failed: [developer-local] => {"failed": true}
msg: Failed to validate the SSL certificate for deb.nodesource.com:443. Make sure your managed systems have a valid CA certificate installed.  If the website serving the url uses SNI you need python >= 2.7.9 on your managed machine.  You can use validate_certs=False if you do not need to confirm the server\s identity but this is unsafe and not recommended Paths checked for this platform: /etc/ssl/certs, /etc/pki/ca-trust/extracted/pem, /etc/pki/tls/certs, /usr/share/ca-certificates/cacert.org, /etc/ansible

FATAL: all hosts have already failed -- aborting

[jwhitlock]

With Ansible 1.9.6 and adding parameter validate_certs: no:

msg: Failed to download key at https://deb.nodesource.com/gpgkey/nodesource.gpg.key: Request failed: <urlopen error [Errno 1] _ssl.c:510: error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure>

Maybe SNI is being used? But Travis CI has a similar error, and it runs Python 2.7.12

[jwhitlock]

And Ansible 2.1.1.0:

fatal: [developer-local]: FAILED! => {"changed": false, "failed": true, "invocation": {"module_args": {"data": null, "file": null, "id": null, "key": null, "keyring": null, "keyserver": null, "state": "present", "url": "https://deb.nodesource.com/gpgkey/nodesource.gpg.key", "validate_certs": true}, "module_name": "apt_key"}, "msg": "Failed to validate the SSL certificate for deb.nodesource.com:443. Make sure your managed systems have a valid CA certificate installed. If the website serving the url uses SNI you need python >= 2.7.9 on your managed machine or you can install the `urllib3`, `pyopenssl`, `ndg-httpsclient`, and `pyasn1` python modules to perform SNI verification in python >= 2.6. You can use validate_certs=False if you do not need to confirm the servers identity but this is unsafe and not recommended. Paths checked for this platform: /etc/ssl/certs, /etc/pki/ca-trust/extracted/pem, /etc/pki/tls/certs, /usr/share/ca-certificates/cacert.org, /etc/ansible"}

I'll experiment with updating Python tomorrow.

[agaffney]

Just another data point...using openssl s_client fails with a handshake failure because the remote side apparently didn't send a cert.

$ openssl s_client -connect deb.nodesource.com:443
CONNECTED(00000003)
140004481599136:error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure:s23_clnt.c:770:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 295 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
---

[mweagle]

Hi @jwhitlock - you're correct, this is a result of us moving the repository hosting to CloudFront. More details are available at nodesource/distributions#353 (comment).

[jwhitlock]

I see a couple of options to make this work with older Python versions (Ubuntu 12.04, 14.04):

1. Switch to installing a key from a file. Add the key file to the repository, such as files/nodesource.gpg.key, and change the task to copy and install from a file::

- name: Copy the NodeSource GPGP key to the remote
  copy:
    src: ../files/nodesource.gpg.key
    dest: /tmp/nodesource.gpg.key

- name: Import the NodeSource GPG key into apt
  apt_key:
    file: /tmp/nodesource.gpg.key
    state: present

1. Add the GPG key on a key server, such https://keyserver.ubuntu.com, and install from there. Something like:

- name: Import the NodeSource GPG key into apt
  apt_key:
    id: 68576280
    url: "https://keyserver.ubuntu.com/pks/lookup?op=get&fingerprint=on&search=0x1655A0AB68576280"

I've uploaded the key at https://keyserver.ubuntu.com/pks/lookup?op=get&fingerprint=on&search=0x1655A0AB68576280

It is also possible that the root page https://deb.nodesource.com could be something other than a redirect and it would work, but that may be a restriction of CloudFront.

[wolfeidau]

Great analysis, interested to hear opinions on using the ubuntu key server rather than nodesource endpoint. cc @jwhitlock @chrislea

Cheers

[jwhitlock]

This fix was integrated into geerlingguy/ansible-role-nodejs@0372961, appears to be working.

[rgagnon24]

I see this happen as well when just attempting to fetch the setup_4.x file from deb.nodesource.com, but oddly only from specific IP addresses.

If I run the command "curl -sL https://deb.nodesource.com/setup_4.x | sudo -E bash -" on Ubuntu 12.04.5 LTS, I get different results depending on the public IP of the server from which I run the command....

I have several machines across Florida, but one of them just cannot get that file, and ALL the machines are running the same version of Ubuntu (all updated), same version of wget, curl, and openssl, but one specific machine just hangs when attempting the above install fetch command.

I get the following when attempting wget, or curl without the pipe to bash:
wget https://deb.nodesource.com/setup_4.x
OpenSSL: error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure
or
curl -sL https://deb.nodesource.com/setup_4.x
error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure

[Hashfyre]

This fix doesn't work anymore. I have started getting the following error consistently:

{
    "changed": false,
    "failed": true,
    "module_stderr": "Traceback (most recent call last):
    File "/tmp/ansible_DPcC4F/ansible_module_apt_repository.py",
    line 522,
    in < module >
    main()
    File "/tmp/ansible_DPcC4F/ansible_module_apt_repository.py",
    line 512,
    in main
    cache.update()
    File "/usr/lib/python2.7/dist-packages/apt/cache.py",
    line 440,
    in update
    raise FetchFailedException(e)
    apt.cache.FetchFailedException: W: Failed to fetch https: //deb.nodesource.com/node_5.x/dists/trusty/main/binary-amd64/Packages  gnutls_handshake() failed: Handshake failed,
        E: Some index files failed to download.They have been ignored,
    or old ones used instead.
    ",
    "module_stdout": "",
    "msg": "MODULE FAILURE",
    "parsed": false
}

[Hashfyre]

Can confirm this is an upstream nodesource issue: nodesource/distributions#388

[mpdude]

Is it that for example on Ubuntu 14.04 apt can not connect over SSL to sites using SNI? It's not just an issue of importing the key but of hitting the repo in general.

[Hashfyre]

ubuntu apt doesn't support SNI, and is listed in the bug-report:
https://bugs.launchpad.net/ubuntu/+source/apt/+bug/1551464

[geerlingguy]

This fix seems to have suddenly stopped working recently :(

[jeffbski]

anybody have any ideas?

The original error mentions that we need python >= 2.7.9 for SNI to work and my current version is 2.7.6 on the server. Would that help anything?

Thanks in advance!

[cesc1989]

Hey, @jeffbski after googling, what Yuri Kanivetsky shares in his answer here https://groups.google.com/forum/#!msg/ansible-project/p4dQ0c25bpM/qSsI4JQqBAAJ helped me.

I needed to make sure these packages are installed: python-urllib3, python-openssl, python-pyasn1, python-pip and installing ndg-httpsclient with pip.

From his answer:

- hosts: all
  tasks:
    - name: Install apt_key dependencies
      apt:
        name: '{{ item }}'
      with_items: [python-urllib3, python-openssl, python-pyasn1, python-pip]
      when: ansible_distribution == 'Ubuntu' or ansible_distribution_release == 'trusty'

    - name: Install apt_key dependencies
      command: pip install ndg-httpsclient
      when: ansible_distribution == 'Ubuntu' or ansible_distribution_release == 'trusty'

[jeffbski]

Thanks @cesc1989 I appreciate it. That seemed to work.