Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ansible randomly thinks apt command failed (either via the apt or the raw modules) #19128

Closed
coaxial opened this issue Dec 9, 2016 · 1 comment
Labels
affects_2.2 This issue/PR affects Ansible v2.2 bug This issue/PR relates to a bug. module This issue/PR relates to a module. python3

Comments

@coaxial
Copy link

coaxial commented Dec 9, 2016

ISSUE TYPE
  • Bug Report
COMPONENT NAME

apt_module, and apt command via the raw_module

ANSIBLE VERSION
ansible 2.2.0.0
  config file = /Users/coaxial/.ansible.cfg
  configured module search path = Default w/o overrides
CONFIGURATION
[defaults]
remote_user = ansible
become_ask_pass = True
OS / ENVIRONMENT

Running on OS X 10.11.6, target host is a minimal/xenial64 box in Vagrant, ansible run via Vagrant's ansible provisioner.

SUMMARY

The apt module seems to think it failed, when it looks like it succeeded. It happens randomly, andI don't know how to reproduce. I also had this issue happen with my bootstrap python task (see below).

STEPS TO REPRODUCE

Run the tasks, and they will randomly fail, maybe once every 20 runs in my case.

---
- name: bootstrap python2
  raw: test -e /usr/bin/python || (apt -qqy update && apt install -qy python-minimal)
  # this task sometimes fails for no reason
  # note to issue readers: I added the options below as a workaround for the task failing, but it's a kludge
  delay: 1
  ignore_errors: yes
  register: task_result
  retries: 5
  until: task_result.rc == 0
---
- name: install essential packages
  apt:
    cache_valid_time: "{{ apt_cache_expire_seconds }}"
    name: "{{ item }}"
    state: present
  with_items:
    - fail2ban
    - htop
    - mosh
    - tmux
    - tree
    - vim
EXPECTED RESULTS

Either the tasks to run without failures, or to fail with an actual failure.

ACTUAL RESULTS

The playbook run fails with the output below. I can't see any actual error in this output, it looks like a successful apt run.

If I ssh into the box, the packages are installed, configured, and running just fine. If I then try to install any other package with apt install foo while sshed into the box, it installs fine too.

If I vagrant destroy -f && vagrant up again, the playbook usually runs fine.

failed: [default] (item=[u'fail2ban', u'htop', u'mosh', u'tmux', u'tree',
u'vim']) => {"cache_update_time": 1481141752, "cache_updated": false, "failed":
true, "item": ["fail2ban", "htop", "mosh", "tmux", "tree", "vim"], "msg":
"'/usr/bin/apt-get -y -o \"Dpkg::Options::=--force-confdef\" -o
\"Dpkg::Options::=--force-confold\"     install 'fail2ban' 'htop' 'mosh' 'tmux'
'tree'' failed: ", "stderr": "", "stdout": "Reading package lists...\nBuilding
dependency tree...\nReading state information...\nThe following additional
packages will be installed:\n  iptables libevent-2.0-5 libnfnetlink0
libprotobuf9v5 libutempter0\n  libxtables11 python3-pyinotify python3-systemd
whois\nSuggested packages:\n  mailx monit strace python-pyinotify-doc\nThe
following NEW packages will be installed:\n  fail2ban htop iptables
libevent-2.0-5 libnfnetlink0 libprotobuf9v5\n  libutempter0 libxtables11 mosh
python3-pyinotify python3-systemd tmux tree\n  whois\n0 upgraded, 14 newly
installed, 0 to remove and 2 not upgraded.\nNeed to get 1612 kB of
archives.\nAfter this operation, 6791 kB of additional disk space will be
used.\nGet:1 http://archive.ubuntu.com/ubuntu xenial/main amd64 libnfnetlink0
amd64 1.0.1-3 [13.3 kB]\nGet:2 http://archive.ubuntu.com/ubuntu xenial/main
amd64 libevent-2.0-5 amd64 2.0.21-stable-2 [113 kB]\nGet:3
http://archive.ubuntu.com/ubuntu xenial/main amd64 libprotobuf9v5 amd64
2.6.1-1.3 [326 kB]\nGet:4 http://archive.ubuntu.com/ubuntu xenial/main amd64
libutempter0 amd64 1.1.6-3 [7898 B]\nGet:5 http://archive.ubuntu.com/ubuntu
xenial/universe amd64 mosh amd64 1.2.5-2 [201 kB]\nGet:6
http://archive.ubuntu.com/ubuntu xenial/main amd64 libxtables11 amd64
1.6.0-2ubuntu3 [27.2 kB]\nGet:7 http://archive.ubuntu.com/ubuntu xenial/main
amd64 iptables amd64 1.6.0-2ubuntu3 [266 kB]\nGet:8
http://archive.ubuntu.com/ubuntu xenial/universe amd64 fail2ban all 0.9.3-1
[227 kB]\nGet:9 http://archive.ubuntu.com/ubuntu xenial-updates/universe amd64
htop amd64 2.0.1-1ubuntu1 [76.4 kB]\nGet:10 http://archive.ubuntu.com/ubuntu
xenial/main amd64 python3-pyinotify all 0.9.6-0fakesync1 [24.7 kB]\nGet:11
http://archive.ubuntu.com/ubuntu xenial/main amd64 python3-systemd amd64
231-2build1 [31.3 kB]\nGet:12 http://archive.ubuntu.com/ubuntu xenial/main
amd64 tmux amd64 2.1-3build1 [223 kB]\nGet:13 http://archive.ubuntu.com/ubuntu
xenial/universe amd64 tree amd64 1.7.0-3 [40.6 kB]\nGet:14
http://archive.ubuntu.com/ubuntu xenial/main amd64 whois amd64 5.2.11 [34.0
kB]\nFetched 1612 kB in 0s (1831 kB/s)\nSelecting previously unselected package
libnfnetlink0:amd64.\r\n(Reading database ... \r(Reading database ...
5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ...
20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database
... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading
database ... 50%\r(Reading database ... 55%\r(Reading database ...
60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database
... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading
database ... 90%\r(Reading database ... 95%\r(Reading database ...
100%\r(Reading database ... 19006 files and directories currently
installed.)\r\nPreparing to unpack .../libnfnetlink0_1.0.1-3_amd64.deb
...\r\nUnpacking libnfnetlink0:amd64 (1.0.1-3) ...\r\nSelecting previously
unselected package libevent-2.0-5:amd64.\r\nPreparing to unpack
.../libevent-2.0-5_2.0.21-stable-2_amd64.deb ...\r\nUnpacking
libevent-2.0-5:amd64 (2.0.21-stable-2) ...\r\nSelecting previously unselected
package libprotobuf9v5:amd64.\r\nPreparing to unpack
.../libprotobuf9v5_2.6.1-1.3_amd64.deb ...\r\nUnpacking libprotobuf9v5:amd64
(2.6.1-1.3) ...\r\nSelecting previously unselected package
libutempter0:amd64.\r\nPreparing to unpack .../libutempter0_1.1.6-3_amd64.deb
...\r\nUnpacking libutempter0:amd64 (1.1.6-3) ...\r\nSelecting previously
unselected package mosh.\r\nPreparing to unpack .../mosh_1.2.5-2_amd64.deb
...\r\nUnpacking mosh (1.2.5-2) ...\r\nSelecting previously unselected package
libxtables11:amd64.\r\nPreparing to unpack
.../libxtables11_1.6.0-2ubuntu3_amd64.deb ...\r\nUnpacking libxtables11:amd64
(1.6.0-2ubuntu3) ...\r\nSelecting previously unselected package
iptables.\r\nPreparing to unpack .../iptables_1.6.0-2ubuntu3_amd64.deb
...\r\nUnpacking iptables (1.6.0-2ubuntu3) ...\r\nSelecting previously
unselected package fail2ban.\r\nPreparing to unpack
.../fail2ban_0.9.3-1_all.deb ...\r\nUnpacking fail2ban (0.9.3-1)
...\r\nSelecting previously unselected package htop.\r\nPreparing to unpack
.../htop_2.0.1-1ubuntu1_amd64.deb ...\r\nUnpacking htop (2.0.1-1ubuntu1)
...\r\nSelecting previously unselected package python3-pyinotify.\r\nPreparing
to unpack .../python3-pyinotify_0.9.6-0fakesync1_all.deb ...\r\nUnpacking
python3-pyinotify (0.9.6-0fakesync1) ...\r\nSelecting previously unselected
package python3-systemd.\r\nPreparing to unpack
.../python3-systemd_231-2build1_amd64.deb ...\r\nUnpacking python3-systemd
(231-2build1) ...\r\nSelecting previously unselected package tmux.\r\nPreparing
to unpack .../tmux_2.1-3build1_amd64.deb ...\r\nUnpacking tmux (2.1-3build1)
...\r\nSelecting previously unselected package tree.\r\nPreparing to unpack
.../tree_1.7.0-3_amd64.deb ...\r\nUnpacking tree (1.7.0-3) ...\r\nSelecting
previously unselected package whois.\r\nPreparing to unpack
.../whois_5.2.11_amd64.deb ...\r\nUnpacking whois (5.2.11) ...\r\nProcessing
triggers for libc-bin (2.23-0ubuntu4) ...\r\nProcessing triggers for systemd
(229-4ubuntu12) ...\r\nProcessing triggers for ureadahead (0.100.0-19)
...\r\nProcessing triggers for mime-support (3.59ubuntu1) ...\r\nSetting up
libnfnetlink0:amd64 (1.0.1-3) ...\r\nSetting up libevent-2.0-5:amd64
(2.0.21-stable-2) ...\r\nSetting up libprotobuf9v5:amd64 (2.6.1-1.3)
...\r\nSetting up libutempter0:amd64 (1.1.6-3) ...\r\nSetting up mosh (1.2.5-2)
...\r\nSetting up libxtables11:amd64 (1.6.0-2ubuntu3) ...\r\nSetting up
iptables (1.6.0-2ubuntu3) ...\r\nSetting up fail2ban (0.9.3-1) ...\r\nSetting
up htop (2.0.1-1ubuntu1) ...\r\nSetting up python3-pyinotify (0.9.6-0fakesync1)
...\r\nSetting up python3-systemd (231-2build1) ...\r\nSetting up tmux
(2.1-3build1) ...\r\nSetting up tree (1.7.0-3) ...\r\nSetting up whois (5.2.11)
...\r\nProcessing triggers for libc-bin (2.23-0ubuntu4) ...\r\nProcessing
triggers for systemd (229-4ubuntu12) ...\r\nProcessing triggers for ureadahead
(0.100.0-19) ...\r\n", "stdout_lines": ["Reading package lists...", "Building
dependency tree...", "Reading state information...", "The following additional
packages will be installed:", "  iptables libevent-2.0-5 libnfnetlink0
libprotobuf9v5 libutempter0", "  libxtables11 python3-pyinotify python3-systemd
whois", "Suggested packages:", "  mailx monit strace python-pyinotify-doc",
"The following NEW packages will be installed:", "  fail2ban htop iptables
libevent-2.0-5 libnfnetlink0 libprotobuf9v5", "  libutempter0 libxtables11 mosh
python3-pyinotify python3-systemd tmux tree", "  whois", "0 upgraded, 14 newly
installed, 0 to remove and 2 not upgraded.", "Need to get 1612 kB of
archives.", "After this operation, 6791 kB of additional disk space will be
used.", "Get:1 http://archive.ubuntu.com/ubuntu xenial/main amd64 libnfnetlink0
amd64 1.0.1-3 [13.3 kB]", "Get:2 http://archive.ubuntu.com/ubuntu xenial/main
amd64 libevent-2.0-5 amd64 2.0.21-stable-2 [113 kB]", "Get:3
http://archive.ubuntu.com/ubuntu xenial/main amd64 libprotobuf9v5 amd64
2.6.1-1.3 [326 kB]", "Get:4 http://archive.ubuntu.com/ubuntu xenial/main amd64
libutempter0 amd64 1.1.6-3 [7898 B]", "Get:5 http://archive.ubuntu.com/ubuntu
xenial/universe amd64 mosh amd64 1.2.5-2 [201 kB]", "Get:6
http://archive.ubuntu.com/ubuntu xenial/main amd64 libxtables11 amd64
1.6.0-2ubuntu3 [27.2 kB]", "Get:7 http://archive.ubuntu.com/ubuntu xenial/main
amd64 iptables amd64 1.6.0-2ubuntu3 [266 kB]", "Get:8
http://archive.ubuntu.com/ubuntu xenial/universe amd64 fail2ban all 0.9.3-1
[227 kB]", "Get:9 http://archive.ubuntu.com/ubuntu xenial-updates/universe
amd64 htop amd64 2.0.1-1ubuntu1 [76.4 kB]", "Get:10
http://archive.ubuntu.com/ubuntu xenial/main amd64 python3-pyinotify all
0.9.6-0fakesync1 [24.7 kB]", "Get:11 http://archive.ubuntu.com/ubuntu
xenial/main amd64 python3-systemd amd64 231-2build1 [31.3 kB]", "Get:12
http://archive.ubuntu.com/ubuntu xenial/main amd64 tmux amd64 2.1-3build1 [223
kB]", "Get:13 http://archive.ubuntu.com/ubuntu xenial/universe amd64 tree amd64
1.7.0-3 [40.6 kB]", "Get:14 http://archive.ubuntu.com/ubuntu xenial/main amd64
whois amd64 5.2.11 [34.0 kB]", "Fetched 1612 kB in 0s (1831 kB/s)", "Selecting
previously unselected package libnfnetlink0:amd64.", "(Reading database ... ",
"(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ...
15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading
database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%",
"(Reading database ... 45%", "(Reading database ... 50%", "(Reading database
... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading
database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%",
"(Reading database ... 85%", "(Reading database ... 90%", "(Reading database
... 95%", "(Reading database ... 100%", "(Reading database ... 19006 files and
directories currently installed.)", "Preparing to unpack
.../libnfnetlink0_1.0.1-3_amd64.deb ...", "Unpacking libnfnetlink0:amd64
(1.0.1-3) ...", "Selecting previously unselected package
libevent-2.0-5:amd64.", "Preparing to unpack
.../libevent-2.0-5_2.0.21-stable-2_amd64.deb ...", "Unpacking
libevent-2.0-5:amd64 (2.0.21-stable-2) ...", "Selecting previously unselected
package libprotobuf9v5:amd64.", "Preparing to unpack
.../libprotobuf9v5_2.6.1-1.3_amd64.deb ...", "Unpacking libprotobuf9v5:amd64
(2.6.1-1.3) ...", "Selecting previously unselected package
libutempter0:amd64.", "Preparing to unpack .../libutempter0_1.1.6-3_amd64.deb
...", "Unpacking libutempter0:amd64 (1.1.6-3) ...", "Selecting previously
unselected package mosh.", "Preparing to unpack .../mosh_1.2.5-2_amd64.deb
...", "Unpacking mosh (1.2.5-2) ...", "Selecting previously unselected package
libxtables11:amd64.", "Preparing to unpack
.../libxtables11_1.6.0-2ubuntu3_amd64.deb ...", "Unpacking libxtables11:amd64
(1.6.0-2ubuntu3) ...", "Selecting previously unselected package iptables.",
"Preparing to unpack .../iptables_1.6.0-2ubuntu3_amd64.deb ...", "Unpacking
iptables (1.6.0-2ubuntu3) ...", "Selecting previously unselected package
fail2ban.", "Preparing to unpack .../fail2ban_0.9.3-1_all.deb ...", "Unpacking
fail2ban (0.9.3-1) ...", "Selecting previously unselected package htop.",
"Preparing to unpack .../htop_2.0.1-1ubuntu1_amd64.deb ...", "Unpacking htop
(2.0.1-1ubuntu1) ...", "Selecting previously unselected package
python3-pyinotify.", "Preparing to unpack
.../python3-pyinotify_0.9.6-0fakesync1_all.deb ...", "Unpacking
python3-pyinotify (0.9.6-0fakesync1) ...", "Selecting previously unselected
package python3-systemd.", "Preparing to unpack
.../python3-systemd_231-2build1_amd64.deb ...", "Unpacking python3-systemd
(231-2build1) ...", "Selecting previously unselected package tmux.", "Preparing
to unpack .../tmux_2.1-3build1_amd64.deb ...", "Unpacking tmux (2.1-3build1)
...", "Selecting previously unselected package tree.", "Preparing to unpack
.../tree_1.7.0-3_amd64.deb ...", "Unpacking tree (1.7.0-3) ...", "Selecting
previously unselected package whois.", "Preparing to unpack
.../whois_5.2.11_amd64.deb ...", "Unpacking whois (5.2.11) ...", "Processing
triggers for libc-bin (2.23-0ubuntu4) ...", "Processing triggers for systemd
(229-4ubuntu12) ...", "Processing triggers for ureadahead (0.100.0-19) ...",
"Processing triggers for mime-support (3.59ubuntu1) ...", "Setting up
libnfnetlink0:amd64 (1.0.1-3) ...", "Setting up libevent-2.0-5:amd64
(2.0.21-stable-2) ...", "Setting up libprotobuf9v5:amd64 (2.6.1-1.3) ...",
"Setting up libutempter0:amd64 (1.1.6-3) ...", "Setting up mosh (1.2.5-2) ...",
"Setting up libxtables11:amd64 (1.6.0-2ubuntu3) ...", "Setting up iptables
(1.6.0-2ubuntu3) ...", "Setting up fail2ban (0.9.3-1) ...", "Setting up htop
(2.0.1-1ubuntu1) ...", "Setting up python3-pyinotify (0.9.6-0fakesync1) ...",
"Setting up python3-systemd (231-2build1) ...", "Setting up tmux (2.1-3build1)
...", "Setting up tree (1.7.0-3) ...", "Setting up whois (5.2.11) ...",
"Processing triggers for libc-bin (2.23-0ubuntu4) ...", "Processing triggers
for systemd (229-4ubuntu12) ...", "Processing triggers for ureadahead
(0.100.0-19) ..."]}
@ansibot ansibot added affects_2.2 This issue/PR affects Ansible v2.2 bug_report module This issue/PR relates to a module. labels Dec 13, 2016
@jimi-c jimi-c removed the plugin label Jan 4, 2017
@ansibot ansibot added the python3 label Feb 7, 2017
abadger added a commit to abadger/ansible that referenced this issue Apr 10, 2017
We invoke /usr/bin/apt inside of the ansible module.  When that command
exits, it doesn't always include a helpful error message.  Include the
exit code so that user's have all the information we can gice them as to
why apt failed.

Addresses ansible#19128
@abadger
Copy link
Contributor

abadger commented Apr 10, 2017

Hello, I've had a chance to look into this and it looks like the problem is most likely in the packages you're trying to install. We only get this particular error message back when /usr/bin/apt-get returns a nonzero exit code to indicate failure. I notice that the last line in the stdout from apt is that it is processing triggers for ureadahead. My guess is that the trigger script sometimes fails and when that happens, it causes apt to fail out of its transaction. I've submitted a PR which will include the exit code from apt in the error message which will at least indicate that Ansible is failing because /usr/bin/apt-get failed. From apt-get's man page, I'm not sure that apt's return code is very useful (it seems to always return 100 on failure) but at least you'll know that something about how apt ran is causing the transient failures.

bcoca pushed a commit that referenced this issue Apr 10, 2017
We invoke /usr/bin/apt inside of the ansible module.  When that command
exits, it doesn't always include a helpful error message.  Include the
exit code so that user's have all the information we can gice them as to
why apt failed.

Addresses #19128
abadger added a commit that referenced this issue Apr 10, 2017
We invoke /usr/bin/apt inside of the ansible module.  When that command
exits, it doesn't always include a helpful error message.  Include the
exit code so that user's have all the information we can gice them as to
why apt failed.

Addresses #19128

(cherry picked from commit b4f976e)
@abadger abadger closed this as completed Apr 11, 2017
abadger added a commit that referenced this issue Apr 13, 2017
We invoke /usr/bin/apt inside of the ansible module.  When that command
exits, it doesn't always include a helpful error message.  Include the
exit code so that user's have all the information we can gice them as to
why apt failed.

Addresses #19128

(cherry picked from commit b4f976e)
abadger added a commit that referenced this issue Apr 13, 2017
We invoke /usr/bin/apt inside of the ansible module.  When that command
exits, it doesn't always include a helpful error message.  Include the
exit code so that user's have all the information we can gice them as to
why apt failed.

Addresses #19128

(cherry picked from commit b4f976e)
@ansibot ansibot added bug This issue/PR relates to a bug. and removed bug_report labels Mar 7, 2018
@ansible ansible locked and limited conversation to collaborators Apr 25, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
affects_2.2 This issue/PR affects Ansible v2.2 bug This issue/PR relates to a bug. module This issue/PR relates to a module. python3
Projects
None yet
Development

No branches or pull requests

4 participants