Can't verify Blue Box apt mirrors

[rlrossiter]

Here's the ansible trace when attempting to run an allinone with Ursula:

TASK: [apt-repos | add any dependent repository keys from url] ****************
Wednesday 28 October 2015  19:22:39 +0000 (0:00:02.021)       0:00:13.959 *****
failed: [allinone] => (item={'repo': u'deb https://apt-mirror.openstack.blueboxgrid.com/bbg_ubuntu/ubuntu precise main', 'key_url': u'https://apt-mirror.openstack.blueboxgrid.com/keys/bbg_ubuntu.key'}) => {"failed": true, "item": {"key_url": "https://apt-mirror.openstack.blueboxgrid.com/keys/bbg_ubuntu.key", "repo": "deb https://apt-mirror.openstack.blueboxgrid.com/bbg_ubuntu/ubuntu precise main"}}
msg: Failed to validate the SSL certificate for apt-mirror.openstack.blueboxgrid.com:443. Use validate_certs=False (insecure) or make sure your managed systems have a valid CA certificate installed. Paths checked for this platform: /etc/ssl/certs, /etc/pki/ca-trust/extracted/pem, /etc/pki/tls/certs, /usr/share/ca-certificates/cacert.org, /etc/ansible
failed: [allinone] => (item={'repo': u'deb https://apt-mirror.openstack.blueboxgrid.com/hwraid/ubuntu precise main', 'key_url': u'https://apt-mirror.openstack.blueboxgrid.com/keys/hwraid.key'}) => {"failed": true, "item": {"key_url": "https://apt-mirror.openstack.blueboxgrid.com/keys/hwraid.key", "repo": "deb https://apt-mirror.openstack.blueboxgrid.com/hwraid/ubuntu precise main"}}
msg: Failed to validate the SSL certificate for apt-mirror.openstack.blueboxgrid.com:443. Use validate_certs=False (insecure) or make sure your managed systems have a valid CA certificate installed. Paths checked for this platform: /etc/ssl/certs, /etc/pki/ca-trust/extracted/pem, /etc/pki/tls/certs, /usr/share/ca-certificates/cacert.org, /etc/ansible
ok: [allinone] => (item={'repo': u'deb https://apt-mirror.openstack.blueboxgrid.com/sensu/apt sensu main', 'key_url': u'https://apt-mirror.openstack.blueboxgrid.com/keys/sensu.key', 'validate_certs': False})

FATAL: all hosts have already failed -- aborting

The task fails to validate the SSL certificates for the blue box apt mirrors. If I wget anything from the mirror, it succeeds. I tried copying the certificate from the mirror into /etc/ssl/certs, but the error still occurred.

I noticed the last repo succeeds, but that is because validate_certs is False.

[epwn]

did you install libssl-dev before running?

[rlrossiter]

@epwn yep, libssl-dev is installed. I just singled out the second repo to test it out. Doing a curl to the repo doesn't have any sort of SSL errors, I can connect to it just fine. So it's only the playbook/ansible that is having problems with this site.

[epwn]

I just re-ran my allinone this morning and didnt hit the issue you are having. So its not reproducible, at least not on my end. I guess you could try (although it wouldnt be best practice) to set validate_certs to false on those two repos and see what happens?

[rlrossiter]

When we set validate_certs to False things worked fine. After looking at https://github.com/ansible/ansible/blob/devel/lib/ansible/module_utils/urls.py#L569, Python 2.7.10 was tried, and that got through without SSL errors. Does there need to be something in the readme to put a requirement on python >= 2.7.9?

[pwplusnick]

@j2sol Should we add a line to requirements.txt that requires python >= 2.7.10 based on this issue?
EDIT: Based on internal chat, we should probably just close this.

[omgjlk]

I'm not sure we want to start expressing python version requirements in requirements.txt. We can document that it is preferred to run Ursula, but we don't know all the ways people install Ursula, and expressing python version requirements there seems antisocial.

[ligc]

HI,

I am seeing a similar issue with the allinone configuration, the first time I run "ursula envs/example/allinone site.yml", it ends with the error below:

TASK: [common | install raid utilities] ***************************************
Wednesday 03 February 2016  04:36:37 -0500 (0:00:00.111)       0:06:37.590 ****
failed: [allinone] => (item=tw-cli,megacli) => {"failed": true, "item": "tw-cli,megacli"}
stderr: E: Failed to fetch https://apt-mirror.openstack.blueboxgrid.com/hwraid/ubuntu/pool-precise/megacli_8.07.14-1_amd64.deb

E: Failed to fetch https://apt-mirror.openstack.blueboxgrid.com/hwraid/ubuntu/pool-precise/tw-cli_2.00.11.020+10.2.1-1_amd64.deb

E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

stdout: Reading package lists...
Building dependency tree...
Reading state information...
The following NEW packages will be installed:
  megacli tw-cli
0 upgraded, 2 newly installed, 0 to remove and 73 not upgraded.
Need to get 5913 kB of archives.
After this operation, 10.4 MB of additional disk space will be used.
Err https://apt-mirror.openstack.blueboxgrid.com/hwraid/ubuntu/ precise/main megacli amd64 8.07.14-1

Err https://apt-mirror.openstack.blueboxgrid.com/hwraid/ubuntu/ precise/main tw-cli amd64 2:2.00.11.020+10.2.1-1


msg: '/usr/bin/apt-get -y -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"   install 'tw-cli' 'megacli'' failed: E: Failed to fetch https://apt-mirror.openstack.blueboxgrid.com/hwraid/ubuntu/pool-precise/megacli_8.07.14-1_amd64.deb

E: Failed to fetch https://apt-mirror.openstack.blueboxgrid.com/hwraid/ubuntu/pool-precise/tw-cli_2.00.11.020+10.2.1-1_amd64.deb

E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?


FATAL: all hosts have already failed -- aborting

If I reran "ursula envs/example/allinone site.yml", then got the same error addressed in this issue topic:

(ursula) root@c910f04x19k02:~/ursula# ursula envs/example/allinone site.yml

PLAY [common role for all hosts] **********************************************
Wednesday 03 February 2016  04:37:36 -0500 (0:00:14.782)       0:00:14.782 ****
===============================================================================

GATHERING FACTS ***************************************************************
Wednesday 03 February 2016  04:37:36 -0500 (0:00:00.000)       0:00:14.782 ****
ok: [allinone]

TASK: [apt-repos | add any dependent repository keys from url] ****************
Wednesday 03 February 2016  04:37:37 -0500 (0:00:00.589)       0:00:15.372 ****
failed: [allinone] => (item={'repo': u'deb https://apt-mirror.openstack.blueboxgrid.com/bbg_ubuntu/ubuntu trusty main', 'key_url': u'https://apt-mirror.openstack.blueboxgrid.com/keys/bbg_ubuntu.key'}) => {"failed": true, "item": {"key_url": "https://apt-mirror.openstack.blueboxgrid.com/keys/bbg_ubuntu.key", "repo": "deb https://apt-mirror.openstack.blueboxgrid.com/bbg_ubuntu/ubuntu trusty main"}}
msg: Failed to validate the SSL certificate for apt-mirror.openstack.blueboxgrid.com:443. Use validate_certs=False (insecure) or make sure your managed systems have a valid CA certificate installed. Paths checked for this platform: /etc/ssl/certs, /etc/pki/ca-trust/extracted/pem, /etc/pki/tls/certs, /usr/share/ca-certificates/cacert.org, /etc/ansible
failed: [allinone] => (item={'repo': u'deb https://apt-mirror.openstack.blueboxgrid.com/hwraid/ubuntu precise main', 'key_url': u'https://apt-mirror.openstack.blueboxgrid.com/keys/hwraid.key'}) => {"failed": true, "item": {"key_url": "https://apt-mirror.openstack.blueboxgrid.com/keys/hwraid.key", "repo": "deb https://apt-mirror.openstack.blueboxgrid.com/hwraid/ubuntu precise main"}}
msg: Failed to validate the SSL certificate for apt-mirror.openstack.blueboxgrid.com:443. Use validate_certs=False (insecure) or make sure your managed systems have a valid CA certificate installed. Paths checked for this platform: /etc/ssl/certs, /etc/pki/ca-trust/extracted/pem, /etc/pki/tls/certs, /usr/share/ca-certificates/cacert.org, /etc/ansible
ok: [allinone] => (item={'repo': u'deb https://apt-mirror.openstack.blueboxgrid.com/sensu/apt sensu main', 'key_url': u'https://apt-mirror.openstack.blueboxgrid.com/keys/sensu.key', 'validate_certs': False})

FATAL: all hosts have already failed -- aborting

I am running Ubuntu 14.04.3 on x86_64 platforms. Should I update Python on the allinone node?

Thank you for any suggestion.

[notnownikki]

That looks like a different issue - perhaps the apt mirror was in the middle of updating, and had got the new package indexes but not all the debs. Can you rerun and see if it completes? Those debs work for me now.

Sorry, I missed the SSL errors at the bottom there. Ignore!

[jzer7]

Guang, I have been having the same error. That is: the Ursula playbook goes through, all apt things work, then changes the repos, then things apt-related fail. As the message from Nov 3 above says, it's due to Ubuntu 14.04-3 LTS using Python 2.7.6, which doesn't come with support for SNI. (Python 2.7.9 adds that).

From what I understand, Ursula uses 2 methods to install packages: (a) the apt module from Ansible with a url, and (b) a url get followed by an install. If you are having a failure of the 1st type, then the hack is to add the "validate_certs: false" to the ansible call. If you are having a failure of the 2nd type (I think your message shows that), then the hack that has worked for me is to create a file (/etc/apt/apt.conf.d/00ursula), that contains the lines:

Acquire::https::apt-mirror.openstack.blueboxgrid.com::Verify-Peer "false";
Acquire::https::file-mirror.openstack.blueboxgrid.com::Verify-Peer "false";
Acquire::https::gem-mirror.openstack.blueboxgrid.com::Verify-Peer "false";
Acquire::https::pypi-mirror.openstack.blueboxgrid.com::Verify-Peer "false";

That hack is based on info from here

[omgjlk]

I'm going to bet that you're using the example defaults-2.0.yml, which has a purposefully broken SSL cert in it. That cert is getting used and put into the system certs file, which in turn makes the whole thing broken, and breaks your ability to validate SSL connections.

https://github.com/blueboxgroup/ursula/blob/master/envs/example/defaults-2.0.yml#L456

[ligc]

Thanks @j2sol, @jzer7, @notnownikki for your suggestions. Here are the steps I tried:

1. update to python 2.10 on the allinone node
2. add the lines suggested by jzer7 to /etc/apt/apt.conf.d/00ursula on the allinone node
3. Updated the cert and private key in defaults-2.0.yml using the cert and key created locally using openssl commands

But these above steps did not help.

I then edited the ./roles/apt-repos/tasks/main.yml, changed the validate_certs line to validate_certs: "no", then the "ursula envs/example/allinone site.yml" goes much much further, but finally failed with another SSL cert verification problem:

TASK: [common | ursula_monitoring directories] ********************************
Wednesday 03 February 2016  23:56:53 -0500 (0:00:00.616)       0:01:28.697 ****
ok: [allinone] => (item=/opt/ursula-monitoring/archive)

TASK: [common | fetch ursula_monitoring when install method is tar] ***********
Wednesday 03 February 2016  23:56:53 -0500 (0:00:00.704)       0:01:29.403 ****
failed: [allinone] => {"failed": true}
msg: Failed to validate the SSL certificate for file-mirror.openstack.blueboxgrid.com:443. Use validate_certs=False (insecure) or make sure your managed systems have a valid CA certificate installed. Paths checked for this platform: /etc/ssl/certs, /etc/pki/ca-trust/extracted/pem, /etc/pki/tls/certs, /usr/share/ca-certificates/cacert.org, /etc/ansible

FATAL: all hosts have already failed -- aborting

PLAY RECAP ********************************************************************
Wednesday 03 February 2016  23:56:54 -0500 (0:00:00.807)       0:01:30.210 ****
===============================================================================
Total tasks: 138
===============================================================================
Slowest 25 Tasks
===============================================================================
common | install lm-sensors package ------------------------------------ 11.39s
common | install sensu plugin pip modules ------------------------------- 1.74s
logging | install logstash ---------------------------------------------- 1.73s
common | install sensu -------------------------------------------------- 1.57s
common | update ca certificates ----------------------------------------- 1.46s
common | implement password controls for pam.d login -------------------- 1.24s
common | install shade for ansible modules ------------------------------ 1.22s
common | Permit SSH ----------------------------------------------------- 1.12s
common | pip install eventlet ------------------------------------------- 1.10s
common | pin alembic to 0.7.0 ------------------------------------------- 1.10s
logging | configure rsyslog --------------------------------------------- 1.02s
common | install raid utilities ----------------------------------------- 0.94s
common | python dependencies -------------------------------------------- 0.94s
common | install system tools ------------------------------------------- 0.94s
common | remove unwanted packages --------------------------------------- 0.94s
common | httplib2 and request Python modules that already use system CA certs --- 0.93s
common | update apt index ----------------------------------------------- 0.93s
common | remove older libruby ------------------------------------------- 0.93s
common | ubuntu apt source list ----------------------------------------- 0.93s
common | (possibly self-signed) ssl cert -------------------------------- 0.92s
common | Install prerequisites for acquiring crash dumps ---------------- 0.92s
common | install ufw ---------------------------------------------------- 0.92s
common | sensu monitoring credentials ----------------------------------- 0.88s
common | openstack cert ------------------------------------------------- 0.85s
common | drop ursula release file --------------------------------------- 0.83s
           to retry, use: --limit @/root/site.retry

allinone                   : ok=111  changed=10   unreachable=0    failed=1

(ursula) root@c910f04x19k02:~/ursula# 

I am new with Ursula and Ansible, any futher suggestion will be highly appreciated.

[pwplusnick]

@ligc: I'm not sure how to solve this issue, but one thing that might be helpful in he future is to run w/ the -vv flag for more verbose output.

edit: Although in this case, it does not appear verbosity is a problem, so you can pretty much ignore this comment.

[omgjlk]

At this point your system's certs file is probably messed up. I'd suggest starting over with a clean OS install and not using the purposefully broken SSL cert in envs/