Mitogen issues in AWX when used verbosity 3 and higher

[Houbovo]

Mitogen is configured to be used in AWX. Version of mitogen: 0.2.2, AWX: 2.0.0
When used verbosity 0 (Normal), 1 (Verbose) or 2 (More verbose) everything works as expected.
When used verbosity 3 (Debug), or 4 (Connection Debug) template runs "forever" with output

[pid 1498] 13:56:51.875998 D mitogen: unix.connect(path='/tmp/mitogen_unix_ahHeVg')
Exception in thread mitogen-broker:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner  self.run()
  File "/usr/lib64/python2.7/threading.py", line 765, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/var/lib/awx/projects/_8__git_onecloud/strategy_plugins/mitogen/core.py", line 340, in <lambda>
    _profile_hook = lambda name, func, *args: func(*args)
  File "/var/lib/awx/projects/_8__git_onecloud/strategy_plugins/mitogen/core.py", line 1813, in _broker_main
    LOG.exception('_broker_main() crashed')
  File "/usr/lib64/python2.7/logging/__init__.py", line 1182, in exception
    self.error(msg, *args, **kwargs)
  File "/usr/lib64/python2.7/logging/__init__.py", line 1175, in error
    self._log(ERROR, msg, args, **kwargs)
  File "/usr/lib64/python2.7/logging/__init__.py", line 1268, in _log
    self.handle(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 1278, in handle
    self.callHandlers(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 1318, in callHandlers
    hdlr.handle(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 749, in handle
    self.emit(record)
  File "/var/lib/awx/projects/_8__git_onecloud/strategy_plugins/ansible_mitogen/logging.py", line 70, in emit
    display.error(s, wrap_text=False)
  File "/usr/lib/python2.7/site-packages/awx/lib/awx_display_callback/display.py", line 40, in wrapper
    with event_context.set_local(**context):
  File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/usr/lib/python2.7/site-packages/awx/lib/awx_display_callback/events.py", line 96, in set_local
    self.remove_local(**kwargs)
  File "/usr/lib/python2.7/site-packages/awx/lib/awx_display_callback/events.py", line 88, in remove_local
    self._local._ctx.pop(key, None)
AttributeError: 'thread._local' object has no attribute '_ctx' 

Steps to reproduce:

1. Install AWX, mitogen, put mitogen in strategy_plugins (everything standard installation)
2. Create some template in AWX
3. Run the template in AWX with desired verbosity

[dw]

It looks like 2 issues here:

• broker_main() crashes, which is probably a Mitogen bug, but
• it can't log the cause of the crash because of the events.py issue, which looks like an AWX bug. I'm not entirely sure what the relevant AWX code is doing, but it's possible Mitogen's use of the Ansible display framework from a non-main thread is confusing it

Please leave this with me and I'll try to setup a reproduction shortly.

[dw]

Re: your offer to supply a public AWX instance, that would be immensely useful! Sorry, just digging through old chat history and noticed you message. :)

[dw]

I've spotted the bug in AWX at least. Going to open a PR for that to begin with.

[dw]

The pull request is here: ansible/awx#2565

Allow me a few hours to think about a workaround I can include in Mitogen, but it looks like potentially it would have to be a monkeypatch.

[dw]

Hi there,

I have checked in a workaround for the issue on dmw branch of the Git repository. Would it be possible for you to test it? I don't know much about AWX configuration :)

I left the worker you supplied unpatched, so dropping the dmw Mitogen branch into its config should be enough to prove the workaround I've added works correctly.

[dw]

Thanks a ton for reporting this! The workaround on master solves the problem, and hopefully future versions of AWX will include the PR I sent.

This is now on the master branch and will make it into the next release. To be updated when a new release is made, subscribe to https://networkgenomics.com/mail/mitogen-announce/

Thanks for reporting this!