Mitogen runner will sometimes mask playbook execution errors

[cloudnull]

While running the openstack-ansible playbooks I ran into the following error: https://pasted.tech/pastes/3cbdf40acfc84ef8311be3475c2cc9636233078e.raw

The task being executed is through a handler which can be found here: https://github.com/logan2211/ansible-haproxy-endpoints/blob/90603684ba258516d10c33f8a01852d59e755d56/handlers/main.yml#L17

This issues was able to be reproduced consistently across playbook runs, however, its not clear to me what the actual issue is?

This is the focused stacktrace

The full traceback is:
Traceback (most recent call last):
  File "/opt/ansible-runtime/local/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 106, in run
    item_results = self._run_loop(items)
  File "/opt/ansible-runtime/local/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 343, in _run_loop
    res = self._execute(variables=task_vars)
  File "/opt/ansible-runtime/local/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 612, in _execute
    result = self._handler.run(task_vars=variables)
  File "/etc/ansible/roles/mitogen/ansible_mitogen/mixins.py", line 117, in run
    return super(ActionModuleMixin, self).run(tmp, task_vars)
  File "/opt/ansible-runtime/local/lib/python2.7/site-packages/ansible/plugins/action/normal.py", line 46, in run
    result = merge_hash(result, self._execute_module(task_vars=task_vars, wrap_async=wrap_async))
  File "/etc/ansible/roles/mitogen/ansible_mitogen/mixins.py", line 346, in _execute_module
    timeout_secs=self.get_task_timeout_secs(),
  File "/etc/ansible/roles/mitogen/ansible_mitogen/planner.py", line 495, in invoke
    kwargs=planner.get_kwargs(),
  File "/etc/ansible/roles/mitogen/ansible_mitogen/connection.py", line 466, in call
    return self._rethrow(recv)
  File "/etc/ansible/roles/mitogen/ansible_mitogen/connection.py", line 452, in _rethrow
    return recv.get().unpickle()
  File "/etc/ansible/roles/mitogen/mitogen/core.py", line 797, in unpickle
    raise obj
CallError: socket.error: [Errno 111] Connection refused
  File "<stdin>", line 3047, in _dispatch_one
  File "master:/etc/ansible/roles/mitogen/ansible_mitogen/target.py", line 423, in run_module
    return impl.run()
  File "master:/etc/ansible/roles/mitogen/ansible_mitogen/runner.py", line 379, in run
    return self._run()
  File "master:/etc/ansible/roles/mitogen/ansible_mitogen/runner.py", line 870, in _run
    self._run_code(code, mod)
  File "master:/etc/ansible/roles/mitogen/ansible_mitogen/runner.py", line 849, in _run_code
    exec('exec code in vars(mod)')
  File "<string>", line 1, in <module>
  File "master:/opt/ansible-runtime/local/lib/python2.7/site-packages/ansible/modules/net_tools/haproxy.py", line 449, in <module>
  File "master:/opt/ansible-runtime/local/lib/python2.7/site-packages/ansible/modules/net_tools/haproxy.py", line 445, in main
  File "master:/opt/ansible-runtime/local/lib/python2.7/site-packages/ansible/modules/net_tools/haproxy.py", line 394, in act
  File "master:/opt/ansible-runtime/local/lib/python2.7/site-packages/ansible/modules/net_tools/haproxy.py", line 325, in get_state_for
  File "master:/opt/ansible-runtime/local/lib/python2.7/site-packages/ansible/modules/net_tools/haproxy.py", line 245, in execute
  File "/usr/lib64/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)


fatal: [utility1_galera_container-9d5dac72]: FAILED! => {
    "msg": "Unexpected failure during module execution.",
    "stdout": ""
}

It makes sense the issue is within the exec in runner.py however I'm simply not sure why.

[cloudnull]

I'm able to recreate this issue using the HEAD of master.

Using commit 79c6d758f3081b7c199e6c8087ecb76cef1eb603 + the Issue510 branch + PR #515

[cloudnull]

Looks like the underlying error is simply being masked. This is what I get when going back to stock, which is likely an environmental issue.

Traceback (most recent call last):
  File "/root/.ansible/tmp/ansible-tmp-1549734302.13-244668050956889/AnsiballZ_haproxy.py", line 113, in <module>
    _ansiballz_main()
  File "/root/.ansible/tmp/ansible-tmp-1549734302.13-244668050956889/AnsiballZ_haproxy.py", line 105, in _ansiballz_main
    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)
  File "/root/.ansible/tmp/ansible-tmp-1549734302.13-244668050956889/AnsiballZ_haproxy.py", line 48, in invoke_module
    imp.load_module('__main__', mod, module, MOD_DESC)
  File "/tmp/ansible_haproxy_payload_hWZRdj/__main__.py", line 449, in <module>
  File "/tmp/ansible_haproxy_payload_hWZRdj/__main__.py", line 445, in main
  File "/tmp/ansible_haproxy_payload_hWZRdj/__main__.py", line 394, in act
  File "/tmp/ansible_haproxy_payload_hWZRdj/__main__.py", line 325, in get_state_for
  File "/tmp/ansible_haproxy_payload_hWZRdj/__main__.py", line 245, in execute
  File "/usr/lib64/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)
socket.error: [Errno 111] Connection refused

[cloudnull]

Just to circle back, once I corrected this environmental issue everything worked as expected.

[dw]

That doesn't look good. Do you think it's Mitogen raising the wrong exception type perhaps? I'll setup a local reproduction, should be easy to fix

[dw]

ooh! It's because the module output doesn't contain the error, right

[dw]

I've seen this a million times and never stopped to think about it -- its because there is nothing in ansible_mitogen/runner.py NewStyleRunner._run() bothering to catch any exception from the module aside from SystemExit. So that propagates back up to the call dispatcher which returns it as a CallError to the worker process, which rethrows it, and again nothing catches CallError so it turns into a crash of the action plug-in.

This is just some missed emulation of vanilla, thanks for reporting it!

[dw]

Thanks for reporting this! If I've misunderstood the problem please don't hesitate to reopen

[cloudnull]

The "issue527" branch seems to have fixed it. Thanks for smashing that so quickly!