Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mitogen does not support Ansible 2.8 yet #587

Closed
opoplawski opened this issue May 12, 2019 · 6 comments

Comments

Projects
None yet
3 participants
@opoplawski
Copy link

commented May 12, 2019

  • Which version of Ansible are you running? 2.8rc3

  • Is your version of Ansible patched in any way? no

  • Are you running with any custom modules, or module_utils loaded? yes

  • Have you tried the latest master version from Git? yes - this is with latest master

  • Mention your host and target OS and versions

Host - EL7.6

  • Mention your host and target Python versions

python 2.7.5

PLAY [webfilter*] **********************************************************************************************************************************************************
[mux  9691] 17:21:18.993407 D mitogen: mitogen.service.Pool(0x7f124f022190, size=32, th='MainThread'): initialized
[mux  9691] 17:21:18.996991 D ansible_mitogen.process: Service pool configured: size=32

TASK [Gathering Facts] *****************************************************************************************************************************************************
task path: /export/home/orion-admin/ansible-boulder/playbooks/e2guardian.yml:2
Sunday 12 May 2019  17:21:19 -0600 (0:00:00.185)       0:00:00.185 ************ 
ERROR! Unexpected Exception, this is probably a bug: expected string or buffer
the full traceback was:

Traceback (most recent call last):
  File "/bin/ansible-playbook", line 110, in <module>
    exit_code = cli.run()
  File "/usr/lib/python2.7/site-packages/ansible/cli/playbook.py", line 123, in run
    results = pbex.run()
  File "/usr/lib/python2.7/site-packages/ansible/executor/playbook_executor.py", line 170, in run
    result = self._tqm.run(play=play)
  File "/usr/lib/python2.7/site-packages/ansible/executor/task_queue_manager.py", line 249, in run
    play_return = strategy.run(iterator, play_context)
  File "/export/home/orion-admin/ansible-boulder/mitogen/ansible_mitogen/strategy.py", line 254, in run
    lambda: run(iterator, play_context)
  File "/export/home/orion-admin/ansible-boulder/mitogen/mitogen/core.py", line 561, in _profile_hook
    return func(*args)
  File "/export/home/orion-admin/ansible-boulder/mitogen/ansible_mitogen/strategy.py", line 254, in <lambda>
    lambda: run(iterator, play_context)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/strategy/linear.py", line 306, in run
    self._queue_task(host, task, task_vars, play_context)
  File "/export/home/orion-admin/ansible-boulder/mitogen/ansible_mitogen/strategy.py", line 229, in _queue_task
    class_only=True,
  File "/export/home/orion-admin/ansible-boulder/mitogen/ansible_mitogen/strategy.py", line 98, in wrap_connection_loader__get
    return connection_loader__get(name, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/loader.py", line 557, in get
    path = self.find_plugin(name, collection_list=collection_list)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/loader.py", line 506, in find_plugin
    plugin = self._find_plugin(name, mod_type=mod_type, ignore_deprecated=ignore_deprecated, check_aliases=check_aliases, collection_list=collection_list)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/loader.py", line 395, in _find_plugin
    if (is_collection_ref(name) or collection_list) and not name.startswith('Ansible'):
  File "/usr/lib/python2.7/site-packages/ansible/utils/collection_loader.py", line 298, in is_collection_ref
    return bool(_collection_qualified_re.match(candidate_name))
TypeError: expected string or buffer
[mux  9691] 17:21:19.052346 D mitogen: Waker(Broker(0x7f124f00fa10) rfd=6, wfd=8).on_disconnect()
[mux  9691] 17:21:19.052735 D mitogen: Router(Broker(0x7f124f00fa10)): stats: 0 module requests in 0 ms, 0 sent (0 ms minify time), 0 negative responses. Sent 0.0 kb total, 0.0 kb avg.                             
@ulrichSchreiner

This comment has been minimized.

Copy link

commented May 17, 2019

this happens with the 2.8 release too

dw added a commit that referenced this issue May 19, 2019

@dw dw pinned this issue May 19, 2019

@dw dw changed the title Traceback with ansible 2.8 rc3 Mitogen does not support Ansible 2.8 yet May 19, 2019

dw added a commit that referenced this issue May 19, 2019

Merge remote-tracking branch 'origin/version-check'
* origin/version-check:
  issue #587: update Changelog
  issue #587: ansible: descriptive version check during startup.

dw added a commit that referenced this issue May 19, 2019

Merge remote-tracking branch 'origin/v027' into stable
* origin/v027:
  docs: update Changelog for release.
  Bump version for release.
  issue #587: update Changelog
  issue #587: ansible: descriptive version check during startup.
  ansible: descriptive version check during startup.
  issue #581: expose mitogen_mask_remote_name variable.
  issue #576: fix Kwargs minor version check.
  issue #574: fix ISSUE_TEMPLATE link
  issue #575: fix exception text rendering
  docs: remove infringing mark
  docs: fix config var scope
  docs: faster stats-preserving redirect
  docs: update ansible page
  issue #570: add firewalld to always-fork list for now.
  docs: removed excess word
  docs: fixed message routing example description
  docs: removed repeated word
  docs: update Changelog; closes #557.
  issue #557: support correct cpu_set_t size

dw added a commit that referenced this issue May 26, 2019

dw added a commit that referenced this issue May 26, 2019

issue #587: 2.8 PlayContext.connection no longer contains connection …
…name

Not clear what the intention is here. Either need to ferret it out of
some other location, or just stop preloading the connection class in the
top-level process.

dw added a commit that referenced this issue May 26, 2019

@dw

This comment has been minimized.

Copy link
Owner

commented May 26, 2019

Some initial work has started in the issue587 branch, but it's far from complete. Summary of the work so far:

  • Support for the obsolete config system has been significantly dropped in 2.8, meaning that Connection plugins and (new) Become plugins are responsible for looking up their own configuration values. This is progress, but it puts Mitogen in a tough situation since we want to maintain support for Ansible 2.3/2.4, where the new config system was either non-existent or in a very immature state.

    I will not drop support for 2.3 since the extension has 2.3 existing users, and newer Ansibles broke the syntax of working configurations.

    It seems the option here is to internally reimplement the new configuration system. That's not a great deal of work, and in fact it's an opportunity to further improve the robustness of connection delegation (by merging the config lookup method for 'direct connect' and 'delegated connection'). I also like this option because upstream's new config system regularly features quite high up in profiler output, and so there is potential to improve on its performance.

  • Support for become plugins. This sticker here is supporting connection delegation -- it must be possible for an intermediary machine to call back into the become plugin API hosted on the master machine. This again, isn't a huge amount of work, but I've been putting it off to avoid reimplementing the whole thing twice.

    There is currently a very large stream-refactor branch that introduces "internally asynchronous" connect. This is a major design change with many benefits, and I'd like to land it first before addressing become plugins, because the become plugin implementation will also need to be asynchronous to match the new implementation.

AFAIK it should be possible to cut a release without become plugin support, and few users will notice the difference, but for now the config system issue is a blocker. It should only take a few days to implement, but I'm short on time for the coming months.

dw added a commit that referenced this issue May 26, 2019

@dw

This comment has been minimized.

Copy link
Owner

commented May 26, 2019

If anyone wants to give issue587 branch a tyre kick - it would be very useful :)

Some internal variables have changed semantic (so CI is still broken), but I overestimated how much compat code has been removed. It's possible this will be a much quicker success than expected

dw added a commit that referenced this issue May 27, 2019

issue #587: mitogen_doas should not become_exe for doas_path
Looks like this has always been wrong - when used as a connection
method, PlayContext.become_method/become_exe may hold totally unrelated
data.

dw added a commit that referenced this issue May 27, 2019

dw added a commit that referenced this issue May 27, 2019

issue #587: "state: absent" was removed in 2.8.0
ansible/cc9c72d6f845710b24e952670b534a57f6948513

dw added a commit that referenced this issue May 27, 2019

dw added a commit that referenced this issue May 27, 2019

dw added a commit that referenced this issue May 27, 2019

dw added a commit that referenced this issue May 27, 2019

dw added a commit that referenced this issue May 27, 2019

dw added a commit that referenced this issue May 27, 2019

dw added a commit that referenced this issue May 27, 2019

Merge remote-tracking branch 'origin/issue587'
* origin/issue587:
  issue #587: docs: update Changelog.
  issue #587: disable SSH key setup, it breaks unit tests
  issue #587: attempt to fix Mac Azure job
  issue #587: Add 2.8.0/Py2.7 job to Azure
  issue #587: use deadsnakes PPA Python because VSTS version is junk
  issue #587: workaround for ansible/ansible#56629
  issue #587: enable spawn_reverse_shell and tidy up Azure step names
  issue #587: ensure Azure worker has a working SSH configuration
  issue #587: import spawn_reverse_shell.py script.
  issue #587: support pausing ansible_tests if flag file exists
  issue #587: "state: absent" was removed in 2.8.0
  issue #587: consistent become_exe() behaviour for older Ansibles.
  issue #587: update stub_connections/ test to use new doas var
  issue #587: update MODULE FAILURE message format for post >2.7
  issue #587: fix syntax error due to presence of comment
  issue #587: update mitogen_doas doc to match varible change.
  issue #587: disable deprecation_warnings for CI.
  issue #587: mitogen_doas should not become_exe for doas_path
  issue #587: 2.8 whitespace handling was improved.
  issue #587: 2.8 PlayContext lacks sudo_flags attribute.
  issue #587: 2.8 PluginLoader.get() introduced new collection_list kwarg
  issue #587: 2.8 PlayContext.connection no longer contains connection name
  issue #587: bump max Ansible version
@dw

This comment has been minimized.

Copy link
Owner

commented May 27, 2019

Bare minimum implementation seems to be done now -- please open fresh issues for anything you bump into, but CI is passing so I'm happy :)


This is now on the master branch and will make it into the next release. To be updated when a new release is made, subscribe to https://www.freelists.org/list/mitogen-announce

Thanks for reporting this!

@dw dw closed this May 27, 2019

@dw dw unpinned this issue May 27, 2019

@opoplawski

This comment has been minimized.

Copy link
Author

commented May 28, 2019

Thanks for working on this. I'm running into #590 now, so following that.

@dw

This comment has been minimized.

Copy link
Owner

commented May 30, 2019

Hi @opoplawski, there is now a workaround for #590 on current master. Note CI is currently only set up for 2.8.0 Py2.7/Py3.5 host -> Py2.7 interpreter, so another variant of #590 may be lurking in the -> Py3.5 interpreter direction

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.