Ziploader "recursive imports" and caching

[abadger]

ISSUE TYPE

• Feature Pull Request

SUMMARY

The first version of ziploader is able to find module_utils files that are listed in the module itself and make sure that those files are included in the zipfile that is sent to the remote machine. This is suboptimal as the equivalent python import will also find the module_utils files that the module_utils files have a need for. This set of commits adds that capability to ziploader and implements a simple controller-side cache to offset the increase in time that searching for the module_utils causes. The cache cleans itself up and auto-invalidates when the main ansible/ansible-playbook process ends.

Caveats to the recusive feature:

• relative imports (py2.6+) are not handled (example: from . import urls). Imports must be absolute imports coming from ansible.module_utils.
• python packages (directories with __init__.py in them). module_utils will only be searched for *.py files right now, not directories. (example: from ansible.module_utils.basic import AnsibleModule will look for ansible/module_utils/basic.py. It will not look for ansible/module_utils/basic/__init__.py even though python itself would.)

There's also a couple unrelated bug fixes (Fixing encoding string in the wrapper which was being lost when we implemented comment stripping of the wrapper and a correction to the process.worker.run documentation since worker no longer loops).

[abadger]

New commits make these changes, also needed for 2.1:

• Pass args via stdin (from the wrapper to the module itself) instead of via the environment. This addresss concerns that there's a maximum size to the environment that we may run into.
• Invoke the module as a script instead of via -m. On python-2.4, -m was implemented very differently and had less features.
  • python-2.4, -m basically means "Look for this python module (not package) in the directories (not zip files or other loaders) of sys.path. If found, run the module as if it were a script. Note that python-2.5 has a -m that allows all of this but is buggy when used against a python package and so python-2.6 removes -m's working with packages: http://stackoverflow.com/questions/4050120/execute-an-installed-python-package-as-a-script which means in python-2.6 and the following directory layout:

|- site-packages
+-|- ansible
  +--- __init__.py
  +--- __main__.py

This does not work: python -m ansible but this does python -m ansible.__main__. (and while the same "works" in pyhton-2.5, it causes some other issues.)

In python-2.7+ they apparently fixed the problems with this so that using -m with a package works.

Anyhow... the fact that python-2.4's -m is just a fancy method of locating a script, since we control where the script gets placed, and since python-2.6 is the first point where we have a -m that we can make do what we want, I decided to just invoke a script for now. When we can say that py2.6 is the lowest version of python we support on the managed node side we can re-add -m support.

[abadger]

Latest version has passed travis in my repo: https://travis-ci.org/abadger/ansible/builds/122534682 Ready for review.

[abadger]

At today's meeting, decided to merge and use devel to test that everything works.