ANSIBLE_COLLECTION_PATH is no longer set by ansible-compat and collections are not found.

[gone-for-coding]

Summary

I'm running ansible-lint using pre-commit. While updating to 6.15.0 pre-commit created a new environment and installed all dependencies freshly.

Ansible-lint now fails to find the collections which are correctly installed by ansible-compat. I have verified the folder in ~/.cache/ansible-compat and they are there. I also noticed that ANSIBLE_COLLECTION_PATH is no longer set.

The special thing about my setup is, that I have no system wide ansible or collection installation, so it all lives within pre-commit/ansible-compat.

I also noticed that, during linting a collection, as a fallback it tried to install the collection itself when something before that failed. After temporarily removing the galaxy.yml this no longer happens.

Issue Type

• Bug Report

OS / ENVIRONMENT

ansible-lint --version
ansible-lint 0.1.dev1 using ansible 2.14.5

Which is 6.15 installed by pre-commit cloning the repo.

• ansible installation method: pre-commit
• ansible-lint installation method: pre-commit

STEPS TO REPRODUCE

docker run --rm -it docker.io/library/python:3.11-slim-bullseye bash

Inside the contain run:

apt update && apt install -y git
pip install pre-commit
git clone https://github.com/gone-for-coding/ansible-lint-minimal.git && cd ansible-lint-minimal
pre-commit run ansible-lint -a -v

Desired Behavior

Linting should work just fine.

Actual Behavior

[INFO] Initializing environment for https://github.com/ansible/ansible-lint.
[INFO] Initializing environment for https://github.com/ansible/ansible-lint:.,ansible-core>=2.13.3.
[INFO] Installing environment for https://github.com/ansible/ansible-lint.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
Ansible-lint.............................................................Failed
- hook id: ansible-lint
- duration: 5.38s
- exit code: 2

INFO     Identified /ansible-lint-minimal as project root due .git directory.
INFO     Running ansible-galaxy collection install -v -r requirements.yml
INFO     Set ANSIBLE_LIBRARY=/root/.cache/ansible-compat/214250/modules:/root/.ansible/plugins/modules:/usr/share/ansible/plugins/modules
INFO     Set ANSIBLE_ROLES_PATH=/root/.cache/ansible-compat/214250/roles:/root/.ansible/roles:/usr/share/ansible/roles:/etc/ansible/roles
INFO     Discovered files to lint using: git -c safe.directory=/ansible-lint-minimal ls-files --cached --others --exclude-standard -z
INFO     Excluded removed files using: git -c safe.directory=/ansible-lint-minimal ls-files --deleted -z
INFO     Executing syntax check on playbook playbook_win.yml (0.41s)
WARNING  Listing 1 violation(s) that are fatal
syntax-check[specific]: couldn't resolve module/action 'ansible.windows.win_regedit'. This often indicates a misspelling, missing collection, or incorrect module path.
playbook_win.yml:8:7


                  Rule Violation Summary
 count tag                    profile rule associated tags
     1 syntax-check[specific] min     core, unskippable

Failed after : 1 failure(s), 0 warning(s) on 3 files.

[ssbarnea]

Only ANSIBLE_COLLECTIONS_PATH is supported in newer versions, not the older name (count the 's' suffixes). It should work, but see that the version reported by you is useless, is the fallback value. How did you install the linter?

[gone-for-coding]

Hi @ssbarnea ,
Thanks for the quick reply. I'm running 6.15.0 installed via pre-commit. As pre-commit clones the repo, the version string is the fallback one.

Please have a look at the minimal example. It is a 100% reproduction. The collections are downloaded as specified in the requirements.yml but it seems, that the ANSIBLE_COLLECTIONS_PATH is no longer set in the latest version of ansible-compat whereas ANSIBLE_LIBRARY and ANSIBLE_ROLES_PATH are correctly redirected to the cache path /root/.cache/ansible-compat/214250.

The essential part is, that when there is no system wide or user wide collection installation which would have been picked up by the default paths. This is basically my workaround. Install the collections in a system path in the container which does the linting.

The issue might be related to the recent changes to the collection path search in ansible-compat (https://github.com/ansible/ansible-compat/blob/2c70ee26d3b04f17084377adbc25102117c10118/src/ansible_compat/runtime.py#L148).

[MarkusTeufelberger]

Same issue here, example output in a github action run: https://github.com/mgit-at/ansible-collection-roles/actions/runs/4806675854/jobs/8554484759?pr=36

[drybjed]

I have the same issue with DebOps project, an example CI output:

https://github.com/debops/debops/actions/runs/4799604148/jobs/8539449676#step:5:1

I managed to tell ansible-lint where the correct collection paths are by setting them in ansible.cfg file in the root of the repository. Since ~/.cache/ansible-compat/ path is created by hashing the path to the repository, I'll handle this by generating ansible.cfg before each test which will point ansible-lint to the correct set of collection paths. It's a convoluted way to do this, but at least it works reliably so far.

[MarkusTeufelberger]

Yeah, this might be a workaround for now to get 6.15.0 running in CI, but I don't really want to distribute ansible.cfg files with a collection. Another workaround might be to set the ANSIBLE_COLLECTIONS_PATH environment variable, but that again might mess with other things (especially when I'm actually currently linting a collection...).

[drybjed]

I'm not sure how feasible is setting a variable, since we need to know the ansible-compat hashed path. It probably could be done locally or in a Makefile, but in GitHub Actions? I'm not sure. Generating ansible.cfg and adding it into .gitignore seems like the most reliable way to do this.

[ssbarnea]

I will be back to this bug and fix it before we make the next release. Now that I see get some extra feedback from you regarding #2802 question. Be sure that you vote there as I did not see a big enough number of votes.

The current magic ~/.cache/... approach seems to cause confusions and also makes ansible behave differently when run by our testing tools and directly. If we would switch to using the current project directory (likely CWD), we could ensure that installed requirements are also accessible to ansible without having to alter the environment variables, as ansible is also looking in current directory for collections.

The only downside that I see with this approach is that people will need to add the collections folder to their .gitignore if they did not already.

[drybjed]

@ssbarnea I'm fine with using .cache/ansible/collections/ for this purpose, or some other similar path. Ignoring .cache/ directory in .gitignore is trivial compared to the hoops we would need go through now. :-)

[gone-for-coding]

For me the question remains: Why do we still set ANSIBLE_LIBRARY and ANSIBLE_ROLES_PATH but no longer ANSIBLE_COLLECTIONS_PATH which seems to be the root cause why it breaks.

What about a two stage approach:

1. Fix the missing ANSIBLE_COLLECTIONS_PATH variable to help people quickly.
2. Pursue a more general approach mentioned here Proposal to make project directory the default place for installing ansible-lint requirements #2802.

[drybjed]

@ssbarnea I was able to handle dependent collections in a "normal" ansible-lint test, but ansible-lint-action@main uses Docker containers and still doesn't see the installed collections. Perhaps adding ~/.cache/ to the list of bind-mounted directories inside of the container could solve this problem.