Non-module files in the module directory conflict with modules; "setup.py" in particular #20702
Comments
ansibot
added
affects_2.3
|
Mentioned this to @gundalow on networktocode.. Here's a downstream in reference to this issue.. networktocode/ntc-ansible#142 |
|
I just noticed this first complained about the setup.cfg, just for completeness of reporting the issue, renaming that with DISABLED- prefix, results in it then finding the setup.py, etc etc.. |
gundalow
removed
the
needs_triage
|
So there is a proposed fix in #20717 However there isn't a short term fix, there is one for in 4 versions time, due to deprecating the existing term that's used in many playbooks |
|
resolved_by_pr #20717 |
|
In looking closer at the issues this raised, I wanted to understand the module loading better. Below you can see output of the instrumentation I added to help follow the code. Doing this has raised a few more questions/concerns about how things get found. If you look at the trace below, you can see that the 'setup' module is searched for in various module classes and then globally. I'm curious about the design goals here, as I would think there would be a desire to have more targeted loading of core modules than this portrays. setup aka gather_facts is in modules/system, but modules/system is not searched in a targeted way? The way modules are searched for today essentially flattens the module namespace. setup.py is just an easy conflict to occur, but there could be many others as the search path grows 2-3 levels deep, etc. Perhaps more parts of the hierarchy should be considered. e.g. if I want to override ansible/modules/system/foo, I should be putting my override in my package/system/foo (or package/modules/system/foo if it were to be really consistent). I think this issue is related to an old issue I saw discussed before where modules were found recursively, and it was found this was an unintended, but depended on behavior by the community. Here are the lines cut from below just related to finding 'setup', e.g. via ActionModule.get(), then ActionModule.find_module(), finally module_loader.find_module(): I did see the built-in alias functionality that already exists PluginLoader(s), however adding an alias for 'setup' has no effect via module_loader in this case. You can see from the bottom of this trace, the last call to find_plugin is called without going through the .get() method, so the alias does not take effect. The other point about below is this is where it finds setup.cfg and bombs out when it realizes it does not have an interpreter. One thought might be that it should notice this earlier and just skip the file, attempting to find a better match. e.g. skip files that don't look like executable scripts/.py files? (in this case it'd find the package's setup.py install script, but still a little more specific.) There are also a number of lookup for other "simple" module names (action, async, poll, meta, smart), that are not found (meta is looked for twice, one is found via ActionModule), not sure what those are about either. |
|
ansibot
added
the
support:core
abadger
changed the title
|
To apply bcoca's answer a little more to the question: This is ansible working as designed. Any file in a module directory can So I assume that the basic idea is to have a git repository with an ansible (1) Only include modules in the git repo. Then checkout the repository into an existing LIBRARY directory. (Files that don't presently conflict with other modules won't usually cause problems but they have the potential to if a real module of that name is created in the future or if someone typos something by mistake) (2) Update the checkout then run setup.py to install the module. I'm assuming that there's a setup.py so that the code can be installed via python setup.py install or pip install or similar. Just run that command to put the code into place. (3) Have a subdirectory in the git repo that is only for the module files. Add the subdirectory holding the modules to the library path. Instead of using a single path in ANSIBLE_LIBRARY and placing the git checkout inside of that path, perform your git checkout to another location and then add the subdirectory of the checkout that holds the modules into ANSIBLE_LIBRARY. (4) Use a symlink. Instead of adding the subdirectory to ANSIBLE_LIBRARY, symlink the subdirectory into the existing library path. Although we can (and plan to in the future once we figure out how the naming will fit into the plans we have for enhancing the whole structure of facts) partially mitigate the problem for the specific case of the setup module, there's not really anything we can do to change the underlying requirement that only module files should live inside of hte module directory. I'll close this ticket now as the underlying technology here is working as expected. |
ISSUE TYPE
COMPONENT NAME
executer.py
setup.py
ANSIBLE VERSION
SUMMARY
When using and developing/improving other modules from 3rd parties it is often easier to keep the working git directory in your module path to streamline development cycle.
However one conflict with ansible prevents doing this with many python projects due to the naming conflict with setup.py. Ansible scans everywhere for the setup.py and could find it (recursively) in the module path, thus finding the wrong setup module. This has also been observed to find "setup.cfg" files as well and attempt to load those, which is also a problematic second issue that alternative non-exec files are being found.
STEPS TO REPRODUCE
Playbook used above:
EXPECTED RESULTS
Ansible should find it's setup module in it's own namespace/directory not in the library path.
ACTUAL RESULTS
The text was updated successfully, but these errors were encountered: