Support for ansible collections import hook

[cognifloyd]

Ansible 2.8.4 + mitogen 0.2.8
Collections installed in ~/.ansible/collections
Playbook and all roles are in a collection.

It looks like the mitogen linear_strategy is not compatible with the collections loader.

Sorry I'm working remotely (to test mitogen with this playbook) on my phone so all I can provide are screenshots of the error. I can look up more details next week.

[dw]

Hi Jacob!

For collections they have implemented a Python import hook. Mitogen needs a little massaging to work for it. I'm busy on another project just now but will come to this real soon, it's a fairly straightforward fix

Thanks for reporting, and apologies for the huge delay

[cognifloyd]

When I took the traceback screenshot, I picked the portion that shows Ansible's PEP 451 import loader for collections. I figured it was a conflict in how that import loader and the mitogen import system worked.

Do you need any more details to diagnose this?

[cognifloyd]

Oh, and no problem on the delay. I have been quite swamped as well!

That said, I am very much looking forward to speeding up a playbook using mitogen. It is in a collection and the entire playbook runs with connection: local as it generates things on behalf of the target host, but only needs to save them locally on the controller. Mitogen's eliminating the python startup time for the repeated modules (copy, template, synchronize, and shell) is my best hope for bringing runtime down.

[mkobel]

I get the same error using mitogen 0.2.9 and ansible 2.9.0, it also occurs without connection: local

Output:

ERROR! Unexpected Exception, this is probably a bug: module 'ansible_collections.ansible.builtin.plugins.action' has no attribute 'ActionBase'

the full traceback was:

Traceback (most recent call last):
  File "/usr/bin/ansible-playbook", line 123, in <module>
    exit_code = cli.run()
  File "/usr/lib/python3.7/site-packages/ansible/cli/playbook.py", line 127, in run
    results = pbex.run()
  File "/usr/lib/python3.7/site-packages/ansible/executor/playbook_executor.py", line 169, in run
    result = self._tqm.run(play=play)
  File "/usr/lib/python3.7/site-packages/ansible/executor/task_queue_manager.py", line 219, in run
    strategy = strategy_loader.get(new_play.strategy, self)
  File "/usr/lib/python3.7/site-packages/ansible/plugins/loader.py", line 552, in get
    self._module_cache[path] = self._load_module_source(name, path)
  File "/usr/lib/python3.7/site-packages/ansible/plugins/loader.py", line 525, in _load_module_source
    spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/moritz/git/ansible/strategy_plugins/mitogen-0.2.9/ansible_mitogen/plugins/strategy/mitogen_linear.py", line 56, in <module>
    import ansible_mitogen.strategy
  File "/home/moritz/git/ansible/strategy_plugins/mitogen-0.2.9/ansible_mitogen/strategy.py", line 43, in <module>
    import ansible_mitogen.mixins
  File "/home/moritz/git/ansible/strategy_plugins/mitogen-0.2.9/ansible_mitogen/mixins.py", line 67, in <module>
    class ActionModuleMixin(ansible.plugins.action.ActionBase):
AttributeError: module 'ansible_collections.ansible.builtin.plugins.action' has no attribute 'ActionBase'

[cognifloyd]

@dw any luck with supporting the collections import hook?

[cognifloyd]

So would the import hook need to be addressed here?
https://github.com/dw/mitogen/blob/d6329f3446af7b120b29656d285d740fdb738ac8/ansible_mitogen/strategy.py#L122

Would the module loader and other plugin type leaders also need to be wrapped? Collections can include any type of plugin (afaik - there may be a few exceptions but I'm not sure).

I see an import hook here, would that need to be adjusted?

I'm also looking at the PluginLoader in ansible to figure this out. I see use of imp for older python and importlib for newer python. I see collection_list and various bits that load plugins from collections.

Can this be addressed directly in ansible_mitogen or will the fix need to touch mitogen internals as well? Maybe in mitogen.core?

[cognifloyd]

@s1113950 You said you're looking at adding support for Ansible 2.9 in #323.
Could you take a look at adding this 2.8 feature (collections) or help me figure out how to do so? I'm a little lost as to where adding collections support would need to be (ansible_mitogen vs other mitogen internals).

I'd love to use mitogen, but I depend on collections support so I haven't been able to test any of my playbooks with mitogen yet.

[s1113950]

Sure yeah I can take a look :)
Do you have a basic playbook that can reproduce the error? I haven't used collections through Ansible before.
What also helped me with interpreter discovery was finding out the PR(s) that landed the feature I was trying to add in Mitogen. I'll try and see if I can find what code added the feature on the Ansible side.

[cognifloyd]

Sure, I'll look up the PRs, get a sample playbook, and post some additional background on collections later today. :) thank you

edit: add todo list

• add background and resources
• post a simple sample playbook + collection

[cognifloyd]

FYI: A lot of work for developing collections actually happened in (an)other repo(s). So if you're looking for older info, you'll want to search for "mazer" which was a fork of the ansible-galaxy tool, but the changes have since been merged back into ansible-galaxy. Again, I'll pull together good resources for you later.

[cognifloyd]

The plan is to move most modules/plugins (esp community maintained modules and network device plugins) out of ansible. Instead of batteries included, you will need to use various content collections that have their own release schedule separate from ansible itself.

FQCN = Fully Qualified Collection Name
FQRN = Fully Qualified Role Name (includes the collection)

Background

• Marketing Buzz: https://www.ansible.com/blog/whats-new-in-red-hat-ansible-engine-2.8
• A few paragraphs on why Ansible is moving things into collections: https://docs.ansible.com/ansible/devel/user_guide/collections_using.html#collections
• A webinar on the move to collections: https://www.ansible.com/resources/webinars-training/collections-future-of-how-ansible-content-is-handled

Official docs

These link to devel version as most up-to-date. Some of this used to be in the ansible-galaxy docs, but since 2.8 they've been moving to docs for ansible itself

• Using collections
• Developing collections
• Installing collections with ansible-galaxy

In Ansible's Code

Primary parts of ansible core code:

• PluginLoader
• AnsibleCollectionLoader The history for devel, 2.9, and 2.8 is very helpful here. This contains the PEP 302/451 import loader for collections (search for import_module and MoudleType to see all the salient parts of the import hook/loader)

Ansible code changes:
Mazer was a fork of ansible-galaxy to develop the concept of "collections". There was also a branch of ansible that added mazer support. Once the concept of collections was fleshed out, the ansible branch got merged into devel, mazer's collections code was merged back into ansible-galaxy, and mazer was deprecated.

• The PR that addded collections subcommand to ansible-galaxy: ansible-galaxy: add collection sub command ansible/ansible#57106
• The PR that actually added collections in core ansible: Collection content loading ansible/ansible#52194
• An proof of concept for the core collections PR that is now obsolete: Mazer role loader: Enable loading roles from ~/.ansible/content/ ansible/ansible#42867
• The PR that swapped mazer doc references for ansible-galaxy: Replace Mazer references with ansible-galaxy ansible/ansible#59487
• A PR that added an ansible integration test that included a collection with a bunch of plugins: First batch of incidental integration tests. ansible/ansible#67717

Collection Contents

Collections can contain roles, modules, actions, and many (in 2.10 all) other plugin types. Playbooks can be included in collections, but the clean support for auto finding them without an absolute path is not baked into ansible yet.

target host-side plugin types that mitogen will probably need to handle in collections:

• fully supported in collections
  • modules
  • module_utils (in 2.8 only import ansible_collections...module_utils... was supported not from ansible_collections...module_utils import ...; 2.9 supports it however due to Fix collection loader builtin handling. ansible/ansible#58897)

All controller-side plugin types can be in a collection with a few caveats (not sure which of these mitogen needs to handle or integrate with):

• fully supported in collections (by frequency/type of plugin):
  • less-common but mitogen definitely touches these:
    • connection (looks like there was a significant fix for loading connection [and at least httpapi if not other] plugins in collections in 2.9 with patch ansible-connection collection plugin loading ansible/ansible#59119 which was backported in 2.8.3)
    • strategy
  • common:
    • action
    • callback
    • inventory
  • common Jinja context (These Jinja context plugins must always use FQCN in playbooks as there is no way to tell what the collections lookup path is since vars don't have metadata on where they are defined):
    • filter
    • test
    • lookup
  • network-device related plugins:
    • cliconf
    • netconf
    • httpapi (httpapi is not strictly only for network devices but was written for them)
  • less-common:
    • terminal
    • shell
    • become (still needs mitogen support, not just inside collections)
• supported in collections with caveats (by frequency):
  • less-common:
    • cache (cache plugins are not supported yet for inventory plugin caches, only for core ansible caches like the fact cache)
  • rare:
    • vars (only in ansible 2.10+)
• supported in collections but not used at playbook/adhoc runtime:
  • doc_fragments

The draft collection shows the layout of a collection with dirs for a lot of plugin types.

[cognifloyd]

Here's a simple playbook + collection where the collection includes roles and a module:
https://github.com/newswangerd/collection_demo
Cool. The author has a YouTube overview of the collection_demo as well: https://www.youtube.com/watch?v=d792W44I5KM
Where he talks about mazer ... replace that with ansible-galaxy collection ...

[cognifloyd]

Here's a collection (for debugging collections lol) that has several kinds of plugins:
https://galaxy.ansible.com/alikins/collection_inspect
It includes sample playbooks, a role, 3 modules, and one of each of these plugin types:

• action
• callback
• filter
• lookup
• module_utils
• modules
• vars

[s1113950]

@cognifloyd thanks for all the info!
I'll give it a detailed look tomorrow PST. At first glance there might be some work that needs to be done in strategy.py, loaders.py, core.py, master.py, and connection.py.