Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Ansible-launcher playbook hangs indefinitely with no progress or logging when ask vault pass enabled in ansible.cfg #61

Closed
jamesmarshall24 opened this issue Mar 18, 2021 · 3 comments
Closed

Ansible-launcher playbook hangs indefinitely with no progress or logging when ask vault pass enabled in ansible.cfg #61

jamesmarshall24 opened this issue Mar 18, 2021 · 3 comments

Comments

@jamesmarshall24
Copy link

jamesmarshall24 commented Mar 18, 2021 •
edited
Loading

Issue

When launching a playbook with ansible-launcher, the interface hangs indefinitely with no logging. Strace shows child process is stuck in "read"

Replication

  1. ANSIBLE_ASK_VAULT_PASS=True ansible-launcher playbook ping.yml --limit localhost -ll debug
  2. Wait a few minutes

Expected Results

Job runs

Observed Result

Hangs indefinitely

Additional Info

python --version
Python 3.6.8

cat /etc/os-release 
NAME="Red Hat Enterprise Linux"
VERSION="8.3 (Ootpa)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="8.3"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Red Hat Enterprise Linux 8.3 (Ootpa)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:8.3:GA"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
REDHAT_BUGZILLA_PRODUCT_VERSION=8.3
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8.3"

Some exact behavior also replicated in Fedora 32 with python 3.8.

strace

strace -p 10575
strace: Process 10575 attached
read(3,

lsof

COMMAND     PID     USER   FD   TYPE DEVICE  SIZE/OFF      NODE NAME
ansible-p 10575 jamarsha  cwd    DIR  253,1        34  92274824 /tmp/tmpcb8bnno0
ansible-p 10575 jamarsha  rtd    DIR  253,1       280       128 /
ansible-p 10575 jamarsha  txt    REG  253,1     11744   5162649 /usr/libexec/platform-python3.6
ansible-p 10575 jamarsha  mem    REG  253,1     65848    201839 /usr/lib64/python3.6/lib-dynload/array.cpython-36m-x86_64-linux-gnu.so
ansible-p 10575 jamarsha  mem    REG  253,1     11560    201855 /usr/lib64/python3.6/lib-dynload/syslog.cpython-36m-x86_64-linux-gnu.so
ansible-p 10575 jamarsha  mem    REG  253,2   7594752 149292490 /home/jamarsha/Projects/winston/lib/python3.6/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so
ansible-p 10575 jamarsha  mem    REG  253,1     88680    421380 /usr/lib64/python3.6/lib-dynload/_socket.cpython-36m-x86_64-linux-gnu.so
ansible-p 10575 jamarsha  mem    REG  253,1     17064    873407 /usr/lib64/python3.6/lib-dynload/_hmacopenssl.cpython-36m-x86_64-linux-gnu.so
ansible-p 10575 jamarsha  mem    REG  253,2     46632 138166488 /home/jamarsha/Projects/winston/lib/python3.6/site-packages/cffi.libs/libffi-806b1a9d.so.6.0.4
ansible-p 10575 jamarsha  mem    REG  253,2    856392  28050734 /home/jamarsha/Projects/winston/lib/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so
ansible-p 10575 jamarsha  mem    REG  253,1      7264    201829 /usr/lib64/python3.6/lib-dynload/_opcode.cpython-36m-x86_64-linux-gnu.so
ansible-p 10575 jamarsha  mem    REG  253,1     33232   4202896 /usr/lib64/libuuid.so.1.3.0
ansible-p 10575 jamarsha  mem    REG  253,1     37024   4203562 /usr/lib64/libffi.so.6.0.2
ansible-p 10575 jamarsha  mem    REG  253,1    129136    201813 /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so
ansible-p 10575 jamarsha  mem    REG  253,1    208616   4202826 /usr/lib64/libtinfo.so.6.1
ansible-p 10575 jamarsha  mem    REG  253,1    300104   4202818 /usr/lib64/libncursesw.so.6.1
ansible-p 10575 jamarsha  mem    REG  253,1     96560    873400 /usr/lib64/python3.6/lib-dynload/_curses.cpython-36m-x86_64-linux-gnu.so
ansible-p 10575 jamarsha  mem    REG  253,1     29040    201853 /usr/lib64/python3.6/lib-dynload/select.cpython-36m-x86_64-linux-gnu.so
ansible-p 10575 jamarsha  mem    REG  253,1     16368    421377 /usr/lib64/python3.6/lib-dynload/_posixsubprocess.cpython-36m-x86_64-linux-gnu.so
ansible-p 10575 jamarsha  mem    REG  253,1     25672    421392 /usr/lib64/python3.6/lib-dynload/termios.cpython-36m-x86_64-linux-gnu.so
ansible-p 10575 jamarsha  mem    REG  253,1     19232    201843 /usr/lib64/python3.6/lib-dynload/fcntl.cpython-36m-x86_64-linux-gnu.so
ansible-p 10575 jamarsha  mem    REG  253,2   1822446  85723957 /home/jamarsha/Projects/winston/lib/python3.6/site-packages/yaml/_yaml.cpython-36m-x86_64-linux-gnu.so
ansible-p 10575 jamarsha  mem    REG  253,1     30632    201841 /usr/lib64/python3.6/lib-dynload/binascii.cpython-36m-x86_64-linux-gnu.so
ansible-p 10575 jamarsha  mem    REG  253,1    115008    201816 /usr/lib64/python3.6/lib-dynload/_datetime.cpython-36m-x86_64-linux-gnu.so
ansible-p 10575 jamarsha  mem    REG  253,1    899352    421393 /usr/lib64/python3.6/lib-dynload/unicodedata.cpython-36m-x86_64-linux-gnu.so
ansible-p 10575 jamarsha  mem    REG  253,1    311032    201818 /usr/lib64/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so
ansible-p 10575 jamarsha  mem    REG  253,1     79896    873408 /usr/lib64/python3.6/lib-dynload/_json.cpython-36m-x86_64-linux-gnu.so
ansible-p 10575 jamarsha  mem    REG  253,1    131360    421376 /usr/lib64/python3.6/lib-dynload/_pickle.cpython-36m-x86_64-linux-gnu.so
ansible-p 10575 jamarsha  mem    REG  253,1     20304    421378 /usr/lib64/python3.6/lib-dynload/_random.cpython-36m-x86_64-linux-gnu.so
ansible-p 10575 jamarsha  mem    REG  253,1     13464    201802 /usr/lib64/python3.6/lib-dynload/_bisect.cpython-36m-x86_64-linux-gnu.so
ansible-p 10575 jamarsha  mem    REG  253,1    110408    421379 /usr/lib64/python3.6/lib-dynload/_sha3.cpython-36m-x86_64-linux-gnu.so
ansible-p 10575 jamarsha  mem    REG  253,1    129304    873397 /usr/lib64/python3.6/lib-dynload/_blake2.cpython-36m-x86_64-linux-gnu.so
ansible-p 10575 jamarsha  mem    REG  253,1    615560   5421808 /usr/lib64/libssl.so.1.1.1g
ansible-p 10575 jamarsha  mem    REG  253,1     47704    201821 /usr/lib64/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so
ansible-p 10575 jamarsha  mem    REG  253,1     59144    421388 /usr/lib64/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so
ansible-p 10575 jamarsha  mem    REG  253,2     46832 137763740 /home/jamarsha/Projects/winston/lib/python3.6/site-packages/markupsafe/_speedups.cpython-36m-x86_64-linux-gnu.so
ansible-p 10575 jamarsha  mem    REG  253,1     52480    421383 /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so
ansible-p 10575 jamarsha  mem    REG  253,1     12976    421387 /usr/lib64/python3.6/lib-dynload/grp.cpython-36m-x86_64-linux-gnu.so
ansible-p 10575 jamarsha  mem    REG  253,1    192016   4203342 /usr/lib64/liblzma.so.5.2.4
ansible-p 10575 jamarsha  mem    REG  253,1     42624    201826 /usr/lib64/python3.6/lib-dynload/_lzma.cpython-36m-x86_64-linux-gnu.so
ansible-p 10575 jamarsha  mem    REG  253,1     74744   4203415 /usr/lib64/libbz2.so.1.0.6
ansible-p 10575 jamarsha  mem    REG  253,1     22824    873398 /usr/lib64/python3.6/lib-dynload/_bz2.cpython-36m-x86_64-linux-gnu.so
ansible-p 10575 jamarsha  mem    REG  253,1     34168    201923 /usr/lib64/python3.6/lib-dynload/zlib.cpython-36m-x86_64-linux-gnu.so
ansible-p 10575 jamarsha  mem    REG  253,1     22808    873406 /usr/lib64/python3.6/lib-dynload/_heapq.cpython-36m-x86_64-linux-gnu.so
ansible-p 10575 jamarsha  mem    REG  253,1 217796128   4202871 /usr/lib/locale/locale-archive
ansible-p 10575 jamarsha  mem    REG  253,1     95400   4202879 /usr/lib64/libz.so.1.2.11
ansible-p 10575 jamarsha  mem    REG  253,1   3154704   4202845 /usr/lib64/libc-2.28.so
ansible-p 10575 jamarsha  mem    REG  253,1   2191808   5754900 /usr/lib64/libm-2.28.so
ansible-p 10575 jamarsha  mem    REG  253,1     23392   4202867 /usr/lib64/libutil-2.28.so
ansible-p 10575 jamarsha  mem    REG  253,1     28968   5754898 /usr/lib64/libdl-2.28.so
ansible-p 10575 jamarsha  mem    REG  253,1    320504   5754906 /usr/lib64/libpthread-2.28.so
ansible-p 10575 jamarsha  mem    REG  253,1   3285544   5563349 /usr/lib64/libpython3.6m.so.1.0
ansible-p 10575 jamarsha  mem    REG  253,1   3071448   5162641 /usr/lib64/libcrypto.so.1.1.1g
ansible-p 10575 jamarsha  mem    REG  253,1    252280   4773589 /usr/lib64/ld-2.28.so
ansible-p 10575 jamarsha  mem    REG  253,2     39888 149292488 /home/jamarsha/Projects/winston/lib/python3.6/site-packages/cryptography/hazmat/bindings/_padding.abi3.so
ansible-p 10575 jamarsha  mem    REG  253,1     26998   8863610 /usr/lib64/gconv/gconv-modules.cache
ansible-p 10575 jamarsha    0u   CHR  136,1       0t0         4 /dev/pts/1
ansible-p 10575 jamarsha    1u   CHR  136,1       0t0         4 /dev/pts/1
ansible-p 10575 jamarsha    2u   CHR  136,1       0t0         4 /dev/pts/1
ansible-p 10575 jamarsha    3u   CHR    5,0       0t0      1041 /dev/tty

ansible_launcher.log

210318002319.271 DEBUG 'ansible_launcher.main' No config file file found
210318002319.271 DEBUG 'ansible_launcher.main' Collection doc cache: path=/home/jamarsha/.cache/ansible_launcher/collection_doc_cache.db
210318002319.271 DEBUG 'ansible_launcher.main' Collection doc cache: current version=0.6.0
210318002319.273 DEBUG 'ansible_launcher.main' ansible-playbook found at /home/jamarsha/Projects/winston/bin/ansible-playbook
210318002319.273 DEBUG 'ansible_launcher.main' Running with container_engine=podman <class 'str'>
210318002319.273 DEBUG 'ansible_launcher.main' Running with execution_environment=False <class 'bool'>
210318002319.273 DEBUG 'ansible_launcher.main' Running with ee_image=quay.io/ansible/ansible-runner:devel <class 'str'>
210318002319.273 DEBUG 'ansible_launcher.main' Running with inventory_columns= <class 'str'>
210318002319.273 DEBUG 'ansible_launcher.main' Running with ide=vim <class 'str'>
210318002319.273 DEBUG 'ansible_launcher.main' Running with logfile=/home/jamarsha/Projects/test/ansible_launcher.log <class 'str'>
210318002319.273 DEBUG 'ansible_launcher.main' Running with loglevel=debug <class 'str'>
210318002319.273 DEBUG 'ansible_launcher.main' Running with no_osc4=False <class 'bool'>
210318002319.273 DEBUG 'ansible_launcher.main' Running with web=False <class 'bool'>
210318002319.273 DEBUG 'ansible_launcher.main' Running with app=playbook <class 'str'>
210318002319.273 DEBUG 'ansible_launcher.main' Running with playbook=/home/jamarsha/Projects/test/ping.yml <class 'str'>
210318002319.273 DEBUG 'ansible_launcher.main' Running with artifact=/home/jamarsha/Projects/test/ping_artifact.json <class 'str'>
210318002319.273 DEBUG 'ansible_launcher.main' Running with inventory=[] <class 'list'>
210318002319.273 DEBUG 'ansible_launcher.main' Running with requires_ansible=True <class 'bool'>
210318002319.273 DEBUG 'ansible_launcher.main' Running with cmdline=['--limit', 'localhost'] <class 'list'>
210318002319.273 DEBUG 'ansible_launcher.main' Running with editor=vi +{line_number} {filename} <class 'str'>
210318002319.273 DEBUG 'ansible_launcher.main' Running with editor_is_console=True <class 'bool'>
210318002319.273 DEBUG 'ansible_launcher.main' Running with share_dir=/home/jamarsha/Projects/winston/share/ansible_launcher <class 'str'>
210318002319.273 DEBUG 'ansible_launcher.main' Running with cache_dir=/home/jamarsha/.cache/ansible_launcher <class 'str'>
210318002319.273 DEBUG 'ansible_launcher.main' Running with collection_doc_cache={} <class 'kvs.KeyValueStore'>
210318002319.273 DEBUG 'ansible_launcher.main' Running with original_command=['--ide', 'vscode', 'playbook', 'ping.yml', '--limit', 'localhost', '-ll', 'debug'] <class 'list'>
210318002319.274 DEBUG 'ansible_launcher.main' Running with parse_and_update=<function parse_and_update at 0x7fcda913a730> <class 'function'>
210318002319.274 DEBUG 'ansible_launcher.actions.explore.16e8.playbook' subaction type is playbook
210318002319.274 DEBUG 'ansible_launcher.actions._runner.run' Runner arg: json_mode:True
210318002319.274 DEBUG 'ansible_launcher.actions._runner.run' Runner arg: quiet:True
210318002319.274 DEBUG 'ansible_launcher.actions._runner.run' Runner arg: event_handler:<bound method PlaybookRunner._event_handler of <ansible_launcher.actions._runner.PlaybookRunner object at 0x7fcda914c320>>
210318002319.274 DEBUG 'ansible_launcher.actions._runner.run' Runner arg: envvars:{'ANSIBLE_CONFIG': '/home/jamarsha/.config/ansible/ansible.cfg'}
210318002319.274 DEBUG 'ansible_launcher.actions._runner.run' Runner arg: cancel_callback:<bound method PlaybookRunner.runner_cancelled_callback of <ansible_launcher.actions._runner.PlaybookRunner object at 0x7fcda914c320>>
210318002319.274 DEBUG 'ansible_launcher.actions._runner.run' Runner arg: finished_callback:<bound method PlaybookRunner.runner_finished_callback of <ansible_launcher.actions._runner.PlaybookRunner object at 0x7fcda914c320>>
210318002319.274 DEBUG 'ansible_launcher.actions._runner.run' Runner arg: cmdline:--limit localhost
210318002319.274 DEBUG 'ansible_launcher.actions._runner.run' Runner arg: inventory:[]
210318002319.274 DEBUG 'ansible_launcher.actions._runner.run' Runner arg: playbook:/home/jamarsha/Projects/test/ping.yml
210318002319.276 DEBUG 'ansible_launcher.actions.explore.16e8._run_runner' runner requested to start
210318002348.272 WARNING 'ansible_launcher.run' Dirty exit, killing the pid

pip freeze for venv

pip freeze | grep -i ansible
ansible==2.9.18
ansible-launcher==0.6.0
ansible-runner @ git+https://github.com/ansible/ansible-runner.git@3925b6b426dca1c30e7fb410b270cd97068458aa
@jamesmarshall24
Copy link
Author

Can't replicate when running as root.

@jamesmarshall24
Copy link
Author

Looks like the issue was with ask vault pass in ansible.cfg. Updated replication steps and issue topic:

ANSIBLE_ASK_VAULT_PASS=True ansible-launcher playbook ping.yml --limit localhost -ll debug

@jamesmarshall24 jamesmarshall24 changed the title Ansible-launcher playbook hangs indefinitely with no progress or logging Ansible-launcher playbook hangs indefinitely with no progress or logging when ask vault pass enabled Mar 25, 2021
@jamesmarshall24 jamesmarshall24 changed the title Ansible-launcher playbook hangs indefinitely with no progress or logging when ask vault pass enabled Ansible-launcher playbook hangs indefinitely with no progress or logging when ask vault pass enabled in ansible.cfg Mar 25, 2021
@cidrblock
Copy link
Collaborator

See #91

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@cidrblock @jamesmarshall24