Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mitogen issues in AWX when used verbosity 3 and higher #400

Closed
Houbovo opened this issue Oct 23, 2018 · 6 comments
Closed

Mitogen issues in AWX when used verbosity 3 and higher #400

Houbovo opened this issue Oct 23, 2018 · 6 comments

Comments

@Houbovo
Copy link

@Houbovo Houbovo commented Oct 23, 2018

Mitogen is configured to be used in AWX. Version of mitogen: 0.2.2, AWX: 2.0.0
When used verbosity 0 (Normal), 1 (Verbose) or 2 (More verbose) everything works as expected.
When used verbosity 3 (Debug), or 4 (Connection Debug) template runs "forever" with output

[pid 1498] 13:56:51.875998 D mitogen: unix.connect(path='/tmp/mitogen_unix_ahHeVg')
Exception in thread mitogen-broker:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner  self.run()
  File "/usr/lib64/python2.7/threading.py", line 765, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/var/lib/awx/projects/_8__git_onecloud/strategy_plugins/mitogen/core.py", line 340, in <lambda>
    _profile_hook = lambda name, func, *args: func(*args)
  File "/var/lib/awx/projects/_8__git_onecloud/strategy_plugins/mitogen/core.py", line 1813, in _broker_main
    LOG.exception('_broker_main() crashed')
  File "/usr/lib64/python2.7/logging/__init__.py", line 1182, in exception
    self.error(msg, *args, **kwargs)
  File "/usr/lib64/python2.7/logging/__init__.py", line 1175, in error
    self._log(ERROR, msg, args, **kwargs)
  File "/usr/lib64/python2.7/logging/__init__.py", line 1268, in _log
    self.handle(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 1278, in handle
    self.callHandlers(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 1318, in callHandlers
    hdlr.handle(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 749, in handle
    self.emit(record)
  File "/var/lib/awx/projects/_8__git_onecloud/strategy_plugins/ansible_mitogen/logging.py", line 70, in emit
    display.error(s, wrap_text=False)
  File "/usr/lib/python2.7/site-packages/awx/lib/awx_display_callback/display.py", line 40, in wrapper
    with event_context.set_local(**context):
  File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/usr/lib/python2.7/site-packages/awx/lib/awx_display_callback/events.py", line 96, in set_local
    self.remove_local(**kwargs)
  File "/usr/lib/python2.7/site-packages/awx/lib/awx_display_callback/events.py", line 88, in remove_local
    self._local._ctx.pop(key, None)
AttributeError: 'thread._local' object has no attribute '_ctx' 

Steps to reproduce:

  1. Install AWX, mitogen, put mitogen in strategy_plugins (everything standard installation)
  2. Create some template in AWX
  3. Run the template in AWX with desired verbosity
@dw
Copy link
Member

@dw dw commented Oct 23, 2018

It looks like 2 issues here:

  • broker_main() crashes, which is probably a Mitogen bug, but
  • it can't log the cause of the crash because of the events.py issue, which looks like an AWX bug. I'm not entirely sure what the relevant AWX code is doing, but it's possible Mitogen's use of the Ansible display framework from a non-main thread is confusing it

Please leave this with me and I'll try to setup a reproduction shortly.

@dw
Copy link
Member

@dw dw commented Nov 1, 2018

Re: your offer to supply a public AWX instance, that would be immensely useful! Sorry, just digging through old chat history and noticed you message. :)

@dw
Copy link
Member

@dw dw commented Nov 1, 2018

I've spotted the bug in AWX at least. Going to open a PR for that to begin with.

@dw
Copy link
Member

@dw dw commented Nov 1, 2018

The pull request is here: ansible/awx#2565

Allow me a few hours to think about a workaround I can include in Mitogen, but it looks like potentially it would have to be a monkeypatch.

@dw
Copy link
Member

@dw dw commented Nov 1, 2018

Hi there,

I have checked in a workaround for the issue on dmw branch of the Git repository. Would it be possible for you to test it? I don't know much about AWX configuration :)

I left the worker you supplied unpatched, so dropping the dmw Mitogen branch into its config should be enough to prove the workaround I've added works correctly.

dw added a commit that referenced this issue Nov 1, 2018
dw added a commit that referenced this issue Nov 1, 2018
@dw
Copy link
Member

@dw dw commented Nov 1, 2018

Thanks a ton for reporting this! The workaround on master solves the problem, and hopefully future versions of AWX will include the PR I sent.

This is now on the master branch and will make it into the next release. To be updated when a new release is made, subscribe to https://networkgenomics.com/mail/mitogen-announce/

Thanks for reporting this!

@dw dw closed this Nov 1, 2018
dw added a commit that referenced this issue Nov 1, 2018
- extra minify tests
- more inline docs migration
- issue #400
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants