Ansible runner issues

[philpep]

Here I sumarize issues related to the ansible backend runner.

Ansible has been available as a "connection backend", meaning the use of ansible python api to execute commands on remote hosts. This have several issues:

• The code is very hard to maintain because ansible doesn't have a stable python API. From https://docs.ansible.com/ansible/latest/dev_guide/developing_api.html "This API is intended for internal Ansible use. Ansible may make changes to this API at any time that could break backward compatibility with older versions of the API. Because of this, external use is not supported by Ansible." We are currently no longer compatible with ansible 2.8

• We are actually violating the license because ansible is GPL and testinfra is not (apache2), so importing ansible in python require the code to be under GPL.

Requirements:

• I want to exclude all solutions including writting code with the ansible python API. Every new release of ansible produce a large amount of work and frustration to me.

• Fully backward compatible, at least for general usage (molecule should not break)

• Should be fast (at least no slower than now).

Possible solutions:

https://github.com/ansible/ansible-runner first appeared to be a possible solution, but after reading the documentation and a bit of code, I think it's actually not providing the feature we need (note that I might have missed something...)

I considered interfacing with ansible cli directly, but this will introduce a terrible performance regression.

I think people using testinfra and ansible use the ansible backend for "convenience", because it can read the inventory, but only some tests actually require ansible (only those using the ansible module). Also I think most people use ansible with either local, ssh or docker connection and testinfra already have such connection backends.

So I propose to:

• Write some code to parse/decode ansible configuration and inventory and convert them to testinfra local, ssh or docker backend. ansible-config and ansible-inventory tools can help here.

• Reimplement the ansible module using ansible cli called in subprocess.

This will drop support for other ansible connection plugins (https://docs.ansible.com/ansible/latest/plugins/connection.html), but I think we can implement them later, we already have some code for kubectl, lxc, winrm.

I already have an POC implementation of this and it works pretty well. I'll submit a PR soon.

Please leave a comment if you want to point me on your use case, I'll be happy to test implementation with your code !

Also special thanks to people that have tried to fix this issue by submitting PRs, especially @jctanner and @ianw , your work helped me a lot for sumarizing this issue and find this solution !

Related to #428 #410 and #401

[jctanner]

@philpep fair enough, but i'm still curious if you could elaborate further on what is missing from runner.