Support for ansible collections import hook

[cognifloyd]

Ansible 2.8.4 + mitogen 0.2.8
Collections installed in ~/.ansible/collections
Playbook and all roles are in a collection.

It looks like the mitogen linear_strategy is not compatible with the collections loader.

Sorry I'm working remotely (to test mitogen with this playbook) on my phone so all I can provide are screenshots of the error. I can look up more details next week.

[dw]

Hi Jacob!

For collections they have implemented a Python import hook. Mitogen needs a little massaging to work for it. I'm busy on another project just now but will come to this real soon, it's a fairly straightforward fix

Thanks for reporting, and apologies for the huge delay

[cognifloyd]

When I took the traceback screenshot, I picked the portion that shows Ansible's PEP 451 import loader for collections. I figured it was a conflict in how that import loader and the mitogen import system worked.

Do you need any more details to diagnose this?

[cognifloyd]

Oh, and no problem on the delay. I have been quite swamped as well!

That said, I am very much looking forward to speeding up a playbook using mitogen. It is in a collection and the entire playbook runs with connection: local as it generates things on behalf of the target host, but only needs to save them locally on the controller. Mitogen's eliminating the python startup time for the repeated modules (copy, template, synchronize, and shell) is my best hope for bringing runtime down.

[mkobel]

I get the same error using mitogen 0.2.9 and ansible 2.9.0, it also occurs without connection: local

Output:

ERROR! Unexpected Exception, this is probably a bug: module 'ansible_collections.ansible.builtin.plugins.action' has no attribute 'ActionBase'

the full traceback was:

Traceback (most recent call last):
  File "/usr/bin/ansible-playbook", line 123, in <module>
    exit_code = cli.run()
  File "/usr/lib/python3.7/site-packages/ansible/cli/playbook.py", line 127, in run
    results = pbex.run()
  File "/usr/lib/python3.7/site-packages/ansible/executor/playbook_executor.py", line 169, in run
    result = self._tqm.run(play=play)
  File "/usr/lib/python3.7/site-packages/ansible/executor/task_queue_manager.py", line 219, in run
    strategy = strategy_loader.get(new_play.strategy, self)
  File "/usr/lib/python3.7/site-packages/ansible/plugins/loader.py", line 552, in get
    self._module_cache[path] = self._load_module_source(name, path)
  File "/usr/lib/python3.7/site-packages/ansible/plugins/loader.py", line 525, in _load_module_source
    spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/moritz/git/ansible/strategy_plugins/mitogen-0.2.9/ansible_mitogen/plugins/strategy/mitogen_linear.py", line 56, in <module>
    import ansible_mitogen.strategy
  File "/home/moritz/git/ansible/strategy_plugins/mitogen-0.2.9/ansible_mitogen/strategy.py", line 43, in <module>
    import ansible_mitogen.mixins
  File "/home/moritz/git/ansible/strategy_plugins/mitogen-0.2.9/ansible_mitogen/mixins.py", line 67, in <module>
    class ActionModuleMixin(ansible.plugins.action.ActionBase):
AttributeError: module 'ansible_collections.ansible.builtin.plugins.action' has no attribute 'ActionBase'

[cognifloyd]

@dw any luck with supporting the collections import hook?

[cognifloyd]

So would the import hook need to be addressed here?
https://github.com/dw/mitogen/blob/d6329f3446af7b120b29656d285d740fdb738ac8/ansible_mitogen/strategy.py#L122

Would the module loader and other plugin type leaders also need to be wrapped? Collections can include any type of plugin (afaik - there may be a few exceptions but I'm not sure).

I see an import hook here, would that need to be adjusted?

I'm also looking at the PluginLoader in ansible to figure this out. I see use of imp for older python and importlib for newer python. I see collection_list and various bits that load plugins from collections.

Can this be addressed directly in ansible_mitogen or will the fix need to touch mitogen internals as well? Maybe in mitogen.core?

[cognifloyd]

@s1113950 You said you're looking at adding support for Ansible 2.9 in #323.
Could you take a look at adding this 2.8 feature (collections) or help me figure out how to do so? I'm a little lost as to where adding collections support would need to be (ansible_mitogen vs other mitogen internals).

I'd love to use mitogen, but I depend on collections support so I haven't been able to test any of my playbooks with mitogen yet.

[s1113950]

Sure yeah I can take a look :)
Do you have a basic playbook that can reproduce the error? I haven't used collections through Ansible before.
What also helped me with interpreter discovery was finding out the PR(s) that landed the feature I was trying to add in Mitogen. I'll try and see if I can find what code added the feature on the Ansible side.

[cognifloyd]

Sure, I'll look up the PRs, get a sample playbook, and post some additional background on collections later today. :) thank you

edit: add todo list

• add background and resources
• post a simple sample playbook + collection

[cognifloyd]

FYI: A lot of work for developing collections actually happened in (an)other repo(s). So if you're looking for older info, you'll want to search for "mazer" which was a fork of the ansible-galaxy tool, but the changes have since been merged back into ansible-galaxy. Again, I'll pull together good resources for you later.

[cognifloyd]

The plan is to move most modules/plugins (esp community maintained modules and network device plugins) out of ansible. Instead of batteries included, you will need to use various content collections that have their own release schedule separate from ansible itself.

FQCN = Fully Qualified Collection Name
FQRN = Fully Qualified Role Name (includes the collection)

Background

• Marketing Buzz: https://www.ansible.com/blog/whats-new-in-red-hat-ansible-engine-2.8
• A few paragraphs on why Ansible is moving things into collections: https://docs.ansible.com/ansible/devel/user_guide/collections_using.html#collections
• A webinar on the move to collections: https://www.ansible.com/resources/webinars-training/collections-future-of-how-ansible-content-is-handled

Official docs

These link to devel version as most up-to-date. Some of this used to be in the ansible-galaxy docs, but since 2.8 they've been moving to docs for ansible itself

• Using collections
• Developing collections
• Installing collections with ansible-galaxy

In Ansible's Code

Primary parts of ansible core code:

• PluginLoader
• AnsibleCollectionLoader The history for devel, 2.9, and 2.8 is very helpful here. This contains the PEP 302/451 import loader for collections (search for import_module and MoudleType to see all the salient parts of the import hook/loader)

Ansible code changes:
Mazer was a fork of ansible-galaxy to develop the concept of "collections". There was also a branch of ansible that added mazer support. Once the concept of collections was fleshed out, the ansible branch got merged into devel, mazer's collections code was merged back into ansible-galaxy, and mazer was deprecated.

• The PR that addded collections subcommand to ansible-galaxy: ansible-galaxy: add collection sub command ansible/ansible#57106
• The PR that actually added collections in core ansible: Collection content loading ansible/ansible#52194
• An proof of concept for the core collections PR that is now obsolete: Mazer role loader: Enable loading roles from ~/.ansible/content/ ansible/ansible#42867
• The PR that swapped mazer doc references for ansible-galaxy: Replace Mazer references with ansible-galaxy ansible/ansible#59487
• A PR that added an ansible integration test that included a collection with a bunch of plugins: First batch of incidental integration tests. ansible/ansible#67717

Collection Contents

Collections can contain roles, modules, actions, and many (in 2.10 all) other plugin types. Playbooks can be included in collections, but the clean support for auto finding them without an absolute path is not baked into ansible yet.

target host-side plugin types that mitogen will probably need to handle in collections:

• fully supported in collections
  • modules
  • module_utils (in 2.8 only import ansible_collections...module_utils... was supported not from ansible_collections...module_utils import ...; 2.9 supports it however due to Fix collection loader builtin handling. ansible/ansible#58897)

All controller-side plugin types can be in a collection with a few caveats (not sure which of these mitogen needs to handle or integrate with):

• fully supported in collections (by frequency/type of plugin):
  • less-common but mitogen definitely touches these:
    • connection (looks like there was a significant fix for loading connection [and at least httpapi if not other] plugins in collections in 2.9 with patch ansible-connection collection plugin loading ansible/ansible#59119 which was backported in 2.8.3)
    • strategy
  • common:
    • action
    • callback
    • inventory
  • common Jinja context (These Jinja context plugins must always use FQCN in playbooks as there is no way to tell what the collections lookup path is since vars don't have metadata on where they are defined):
    • filter
    • test
    • lookup
  • network-device related plugins:
    • cliconf
    • netconf
    • httpapi (httpapi is not strictly only for network devices but was written for them)
  • less-common:
    • terminal
    • shell
    • become (still needs mitogen support, not just inside collections)
• supported in collections with caveats (by frequency):
  • less-common:
    • cache (cache plugins are not supported yet for inventory plugin caches, only for core ansible caches like the fact cache)
  • rare:
    • vars (only in ansible 2.10+)
• supported in collections but not used at playbook/adhoc runtime:
  • doc_fragments

The draft collection shows the layout of a collection with dirs for a lot of plugin types.

[cognifloyd]

Here's a simple playbook + collection where the collection includes roles and a module:
https://github.com/newswangerd/collection_demo
Cool. The author has a YouTube overview of the collection_demo as well: https://www.youtube.com/watch?v=d792W44I5KM
Where he talks about mazer ... replace that with ansible-galaxy collection ...

[cognifloyd]

Here's a collection (for debugging collections lol) that has several kinds of plugins:
https://galaxy.ansible.com/alikins/collection_inspect
It includes sample playbooks, a role, 3 modules, and one of each of these plugin types:

• action
• callback
• filter
• lookup
• module_utils
• modules
• vars

[s1113950]

@cognifloyd thanks for all the info!
I'll give it a detailed look tomorrow PST. At first glance there might be some work that needs to be done in strategy.py, loaders.py, core.py, master.py, and connection.py.

[s1113950]

(.venv) steven@Stevens-MBP:~/repos/mitogen-test$ ansible-galaxy collection install alikins.collection_inspect
- downloading role 'collection', owned by
[WARNING]: - collection was NOT installed successfully: Content has no field named 'owner'

ERROR! - you can use --ignore-errors to skip failed roles and finish processing the list.

@cognifloyd I'm probably doing something wrong 🤔 This is with ansible 2.8.8. The link you posted https://galaxy.ansible.com/alikins/collection_inspect said that it's only supported in ansible 2.9+ but you mentioned it was also backported to 2.8.3 so I thought it would work with 2.8.8 as well? How do I install a collection in 2.8?

[s1113950]

Or is it better to just get it to work in 2.9?

[cognifloyd]

doh. I forgot about that. ansible-galaxy didn't get support for installing collections until 2.9. I started using collections with 2.8, but to install them I had a separate virtualenv with the devel version of ansible that I only used for installing the collection.

Yeah, skip 2.8 for now since we can't really install collections with it.

[cognifloyd]

So, I guess that means get everything else working with 2.9 before tackling collections?

[s1113950]

Makes sense to me yeah 👍
I'm gonna update Mitogen's tests with ansible 2.9 and go from there 😄

[s1113950]

Gonna try collections now

[s1113950]

@cognifloyd I found something interesting: I didn't get the actionbase error when I ran the entire collection example from the playbook level, but I did when running it from inside a role at the task level.

ansible-playbook -b plays/collections.yml worked for me when the collections.yml was this:

- name: Test alikins.collection_inspect vars plugin
  hosts: localhost
  gather_facts: false
  vars:
    foo: "fooFooFOO"
    # Test having the filter plugin loaded early
    blip: "{{ foo |alikins.collection_inspect.collection_inspect }}"
  tasks:
    - name: "Test collection_inspect module and module_utils loaded from alikins.collection_inspect collection"
      alikins.collection_inspect.collection_inspect:
        args:
      register: module_result

    - name: Show collection_inspect module results
      debug:
        var: module_result

    # TODO: not sure if vars plugins are expected to work yet
    - name: show vars plugins for collection_inspect
      debug:
        msg: "{{ vars|to_nice_json }}"

I'm gonna run it from the task-level now and get it working there!

[s1113950]

I don't get the error on the task-level when using:
Mitogen: master
Ansible: 2.9.6
collection installed via: .venv/bin/ansible-galaxy collection install alikins.collection_inspect.

Here's the playbook:

- hosts: node
  gather_facts: false
  vars:
    foo: "fooFooFOO"
    # Test having the filter plugin loaded early
    blip: "{{ foo |alikins.collection_inspect.collection_inspect }}"
  roles:
  - run_test

and here's the role:

- name: "Test collection_inspect module and module_utils loaded from alikins.collection_inspect collection"
  alikins.collection_inspect.collection_inspect:
    args:
  register: module_result

- name: Show collection_inspect module results
  debug:
    var: module_result

# TODO: not sure if vars plugins are expected to work yet
- name: show vars plugins for collection_inspect
  debug:
    msg: "{{ vars|to_nice_json }}"

Previously I got the actionbase error when I hadn't removed the following from the original example:
tasks
vars
gather_facts
hosts

I moved them to be declared in the playbook, and then had the playbook run the example there and it worked. @cognifloyd is that not how people run collections? 🤔

[s1113950]

I'll continue trying the other playbooks here: https://github.com/alikins/collection_inspect/tree/master/playbooks

[s1113950]

aHa! Finally able to repro at the task level. Sweet

[s1113950]

The full traceback is:
Traceback (most recent call last):
  File "master:/private/tmp/mitogen/ansible_mitogen/runner.py", line 975, in _run
    self._run_code(code, mod)
  File "master:/private/tmp/mitogen/ansible_mitogen/runner.py", line 941, in _run_code
    exec('exec code in vars(mod)')
  File "<string>", line 1, in <module>
  File "master:/Users/steven/.ansible/collections/ansible_collections/alikins/collection_inspect/plugins/modules/get_collection_inspect.py", line 24, in <module>
ImportError: No module named alikins.collection_inspect.plugins.module_utils.collection_inspect
[WARNING]: Platform darwin on host localhost is using the discovered Python interpreter at /usr/bin/python, but future installation of another Python interpreter
could change this. See https://docs.ansible.com/ansible/2.9/reference_appendices/interpreter_discovery.html for more information.
fatal: [localhost]: FAILED! => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python"
    },
    "changed": false,
    "module_stderr": "Traceback (most recent call last):\n  File \"master:/private/tmp/mitogen/ansible_mitogen/runner.py\", line 975, in _run\n    self._run_code(
code, mod)\n  File \"master:/private/tmp/mitogen/ansible_mitogen/runner.py\", line 941, in _run_code\n    exec('exec code in vars(mod)')\n  File \"<string>\", lin
e 1, in <module>\n  File \"master:/Users/steven/.ansible/collections/ansible_collections/alikins/collection_inspect/plugins/modules/get_collection_inspect.py\", l
ine 24, in <module>\nImportError: No module named alikins.collection_inspect.plugins.module_utils.collection_inspect\n",
    "module_stdout": "",
    "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error",
    "rc": 1
}

currently stuff loads, sorta. Submodules of ansible_collections are empty so the parent has nothing to send to the child in this case.

It has to do with the quite hacky way that collections are assembled (they're effectively a fake module with no real __file__ (it's called __synthetic__ 😂 ). The collections loader also has no nice way of generating source code so the parent is having a hard time sending stuff to the child on import. I think I'm close though 🤔 It might involve a bit of reverse-engineering the loader to "unload" the source on demand, but I have a few other things to try before it gets to that point.

[nitzmahone]

Just a hint: the Python loader underwent a significant rewrite for 2.10 in ansible/ansible#67684 - largely to make it more "Python standards-compliant". That said, we purposely don't "Python load" module/module_util code during Ansible module construction, because that causes the Python module global code to execute in the controller (which can have ... interesting side effects, depending on the module). Unfortunately, pkgutil.get_data still runs package init code (there's an open proposal to extend it not to do so, but alas), so we can't rely exclusively on the Python standard mechanisms for accessing blob data in collections. I could blacklist those locations in the Python loader so they can't run package init code, but that would prevent legit package init code from running for module_utils that are used by controller plugins. So there will likely be a non-importing version of pkgutil.get_data in 2.10... I'd be careful about relying on any of the implementation details of the module_utils location/construction (eg CollectionModuleInfo I saw mentioned in the collections support PR)- those have changed quite a bit, and will probably continue to.