Make strategy_plugins obsolete

[varac]

I'm super impressed by the speedup of ansible_mitogen !
However, there's one thing blocking the rollout in our opensource project, and this is the fact that you need to specify the location of the ansible_mitogen strategy plugin in ansible.cfg.
The location will veary depending of the type of installation, i.e. some will install it under ~/.local/lib/python3.6, some under /user/local/lib/python3.7, some in a virtualenv etc.

We like to ship a ansibile.cfg that doesn't need to get modified manually, so I wonder if there's any change that ansible will try to find the plugin path by itself ? That would make it even easier for the user.

[dw]

Hi :) The simplest solution is to avoid installing via pip. The installation steps avoid pip for this very reason -- the package directory can be placed in any path (e.g. in the same dir as ansible.cfg), and the first loaded plugin fixes the Python path up internally to cope with the install location.

Can't you ship the extension and say, place it in the same directory as ansible.cfg? That way, 'strategy_plugins = mitogen-1.2.3/ansible_mitogen/plugins/strategy' "should" always succeed.

There are a bunch more hooks where it might be possible to trick it on to the Ansible loader path, but I'd need to understand more about your project first in order to suggest one

[varac]

Sure, we can do that, it's just that we already ship a requirements.txt with multiple python deps that we install via pip, and therefore it feel weird to ship a bundled version of one exceptional python module. We'd like to separate our code from external modules.

[dw]

Totes understand the desire to use pip :) The trouble is that Ansible's plugin loader paths are completely separate.

There is at least one interpreter-level hack that is possible to cure this "once and for all", but making global changes to the interpreter config is a very nasty thing for any Python package to do.

[varac]

Ok, it's not a big deal, I wanted to check if this would be easily solveable. We'll probably ship ansible_mitogen for now, until this has been implemented. Thx for the quick response!

[dw]

Thanks for reporting it anyway -- this is a known usability issue and there wasn't an existing ticket for it

[dmsimard]

@dw @varac we had similar challenges getting users to configure the ARA callback plugin :)

Once upon a time, we had users do something like this:

export ara_location=$(python -c "import os,ara; print(os.path.dirname(ara.__file__))")
export ANSIBLE_CALLBACK_PLUGINS=$ara_location/plugins/callbacks

In a script it's not too bad but we needed something better for humans.

We now have python modules that try to be helpful:

$ python -m ara.setup.path
/usr/lib/python2.7/site-packages/ara

$ python -m ara.setup.action_plugins
/usr/lib/python2.7/site-packages/ara/plugins/actions

$ python -m ara.setup.callback_plugins
/usr/lib/python2.7/site-packages/ara/plugins/callbacks

$ python -m ara.setup.ansible | tee ansible.cfg
[defaults]
callback_plugins=/usr/lib/python2.7/site-packages/ara/plugins/callbacks
action_plugins=/usr/lib/python2.7/site-packages/ara/plugins/actions

$ python -m ara.setup.env
export ANSIBLE_CALLBACK_PLUGINS=/usr/lib/python2.7/site-packages/ara/plugins/callbacks
export ANSIBLE_ACTION_PLUGINS=/usr/lib/python2.7/site-packages/ara/plugins/actions

(More info in docs: https://ara.readthedocs.io/en/stable/configuration.html)

The approach is super simple and it's open source 🔥 https://github.com/openstack/ara/tree/master/ara/setup

Somewhere in my to-do list, I need to experiment with putting things in ~/.ansible. Apparently you can put stuff there.

[dw]

@dmsimard I keep forgetting we have the same problem! My main gripe with a wrapper script is that it kicks the bucket down the road.. installation is no longer a one-step process, but something that impacts the interface to the tool, something the Ansible extension works very hard to avoid everywhere else.

I also wonder how ARA's procedure works with AWX. Does it require a different procedure?

Options seem to be:

• Confuse users as today with a PyPI URL but recommend they don't use pip. (This provably isn't working out)
• Wrapper script that changes the interface
• Helper script that modifies the config ("python -m ansible_mitogen.install /path/to/ansible.cfg")
• Mess with the system globally somehow
  • Magic setup.py that installs to ~/.ansible or /usr/share/ansible
  • Magic setup.py that installs a magical .pth file that does even more evil things

Problem with the magic setup.py scripts is that the user has no idea some venv-specific extension version just updated their homedir or the whole system

What I'd really love is for ansible-galaxy install mitogen to just magically work. Looked at it a couple of times, but always inconclusive whether it's possible or not

[dmsimard]

Enabling ARA on AWX is a bit more involved because of a bug around the default virtualenv inheriting from an Ansible installed by RPM: ansible/awx#1737.

I never got to the bottom of it but the workaround is:

• Create a virtualenv that doesn't inherit system-site-packages in /var/lib/awx/venv/foo
• /var/lib/awx/venv/foo/bin/pip install ansible ara
• Add the (system-wide?) callback plugin in the settings UI:
• When configuring job templates, you'll need to make sure you are using the new virtualenv.

Re: options

• Confuse users as today with a PyPI URL but recommend they don't use pip. (This provably isn't working out)

I actually recommend users to install by pip inside a virtualenv. Can you expand on what is confusing with pypi ?

• Wrapper script that changes the interface

To clarify, do you mean something like mitogen -i hosts playbook.yml instead of ansible-playbook -i hosts playbook.yml ? I've been avoiding this option so far.

• Helper script that modifies the config ("python -m ansible_mitogen.install /path/to/ansible.cfg")
• Mess with the system globally somehow
  • Magic setup.py that installs to ~/.ansible or /usr/share/ansible
  • Magic setup.py that installs a magical .pth file that does even more evil things

I'll report back if I get the chance to experiment on ~/.ansible

What I'd really love is for ansible-galaxy install mitogen to just magically work. Looked at it a couple of times, but always inconclusive whether it's possible or not.

Let me know if you end up figuring that out, I'm interested :D

[asfaltboy]

I've noticed this role on galaxy, could this be what you're looking for?

https://github.com/robertdebock/ansible-role-mitogen

I for one, would also love to have Mitogen somewhere easily installable (e.g on pypi) for easier installation on machines that will run the ansible playbooks...

[lassizci]

git subtree would be one option.. although git lfs bre.aks that

[dw]

@dmsimard it looks like the final word on this is Mazer, but I think I've quickly played around twice now and failed to make a working package. It's possible Ansible 2.8 doesn't quite implement the design intent of collections yet.

At least, you can see from the Mazer docs that strategy plugins appear in the example collection layout, but ansible-config dump'does not reveal an installed collection on the usual strategy plugins path, and doing something like collections: - dw.mitogen with strategy: mitogen_linear appears to have no effect.

I'm probably just noobing collections, but it looks like at least in principle the right answer to this ticket