ANSIBLE_ROLES_PATH not set correctly? #3404

sdarwin · 2022-01-18T22:32:40Z

Issue Type

Bug report

Molecule and Ansible details

ansible --version && molecule --version

5.1.0 and 3.5.1

Molecule installation method (one of):

source
pip

Have tried both methods.

Ansible installation method (one of):

source
pip
OS package

pip

Desired Behavior

Molecule tests that have been passing, should continue to be passing. When running in CI, if roles are installed, the ANSIBLE_ROLES_PATH points to the roles.

Actual Behaviour

Running molecule ansible tests in CircleCI.

Certain tests that succeeded one month ago are failing now, even though nothing apparently changed. It's not impossible, that this is being caused by something else entirely such as Ansible itself or another package. However at the moment it looks like molecule so your feedback would be helpful to point in the right direction.

Case 1:
https://app.circleci.com/pipelines/github/cloudalchemy/ansible-prometheus?branch=master&filter=all

When molecule ran on Dec 27, 2021, it succeeded.
Here is a interesting snippet from the log output:

INFO     Added ANSIBLE_ROLES_PATH=/home/circleci/.cache/ansible-compat/dda4ac/roles:/home/circleci/.ansible/roles:/usr/share/ansible/roles:/etc/ansible/roles
INFO     Using /home/circleci/.ansible/roles/cloudalchemy.prometheus symlink to current repository in order to enable Ansible to find the role using its expected full name.

Notice that it's creating a symlink.

Now,

INFO     Set ANSIBLE_ROLES_PATH=/home/circleci/.cache/ansible-compat/dda4ac/roles:/home/circleci/.ansible/roles:/usr/share/ansible/roles:/etc/ansible/roles

The wording changed from "Added" to "Set". So, something changed. And it does not mention the symlink anymore. Probably no symlink is being created. Because of that the test fails, during "molecule syntax". It can't find the role.

Case 2:

This repo used to succeed, about a month ago.
https://app.circleci.com/pipelines/github/prometheus/demo-site/833/workflows/39439543-5173-4027-8be0-3d1817db7b7c/jobs/3397

Now it's failing:
https://app.circleci.com/pipelines/github/prometheus/demo-site/857/workflows/82accec5-9ae4-4c17-ba50-9c25d35b550e/jobs/3516

When does it fail? During the tests, it gets to this step:

run molecule --debug syntax

Again, it can't find the roles. Similar to Case 1.

Where are the roles being installed? It can be seen that ansible-galaxy installs roles in /home/circleci/project/roles/

- extracting cloudalchemy.prometheus to /home/circleci/project/roles/cloudalchemy.prometheus

If we test with ansible 5.1 and molecule version 3.4.1, the tests succeed. Here is log output regarding ANSIBLE_ROLES_PATH

INFO     Added ANSIBLE_ROLES_PATH=~/.ansible/roles:/usr/share/ansible/roles:/etc/ansible/roles:roles
ANSIBLE_ROLES_PATH: /home/circleci/.cache/molecule/project/default/roles:/home/circleci:/home/circleci/.ansible/roles:/usr/share/ansible/roles:/etc/ansible/roles:~/.ansible/roles:/usr/share/ansible/roles:/etc/ansible/roles:roles
ANSIBLE_ROLES_PATH=/home/circleci/.cache/molecule/project/default/roles:/home/circleci:/home/circleci/.ansible/roles:/usr/share/ansible/roles:/etc/ansible/roles:~/.ansible/roles:/usr/share/ansible/roles:/etc/ansible/roles:roles

Next, test with ansible 5.1 and molecule 3.5.1. This fails.

The wording has changed from "Added" to "Set". Why? And, more importantly, the "Set" did not accomplish anything because it didn't set the path.

The debug information (lines 3 and 4) shows something different from the "Set" (line 1).

INFO     Set ANSIBLE_ROLES_PATH=/home/circleci/.cache/ansible-compat/dda4ac/roles:roles:/home/circleci/project/roles
DEBUG: ANSIBLE ENVIRONMENT:
ANSIBLE_ROLES_PATH: '/home/circleci/.cache/molecule/project/default/roles:/home/circleci:/home/circleci/.ansible/roles:/usr/share/ansible/roles:/etc/ansible/roles:'
ANSIBLE_ROLES_PATH=/home/circleci/.cache/molecule/project/default/roles:/home/circleci:/home/circleci/.ansible/roles:/usr/share/ansible/roles:/etc/ansible/roles: ANSIBLE_VAULT_PASSWORD=******* MOLECULE_DEBUG=True

The text was updated successfully, but these errors were encountered:

ssbarnea · 2022-01-19T13:26:14Z

Try the code from main branch and repot the outcome. I am aware of several changes that are likely to affect role paths.

sdarwin · 2022-01-19T14:06:00Z

Tested with main branch. Same result.

3.4.1 is successful. Latest isn't.

To summarize the issues from yesterday:
In Case 2, it seems to indicate that it's "setting" the ANSIBLE_ROLES_PATH to the correct path. But then the debug information that follows is different...
In Case 1, it used to create a symlink to the role, so the role was definitely in the path. And now it doesn't mention a symlink anymore.

sdarwin · 2022-01-19T19:14:42Z

More info: it looks like these changes first appeared 8 days ago and correspond to the release of ansible-compat 1.0.0, 8 days ago. Before the jobs were using Molecule 3.5.2 and ansible-compat-0.5.0 successfully.

ssbarnea · 2022-01-19T20:47:12Z

that is indeed useful. i will try to look at it tomorrow, ping me in irc/matrix if you can and maybe we can debug it and make a hotfix.

sdarwin · 2022-01-19T22:20:55Z

Also it's feasible to replicate the issue pretty quickly by forking these repos and enabling circleci builds:

https://github.com/cloudalchemy/ansible-prometheus 
https://github.com/prometheus/demo-site

ansible-compat/issues/114 looks similar. Could be identical.

artis3n · 2022-01-20T03:28:07Z

This is occurring on https://github.com/artis3n/ansible-role-tailscale as well after updating to ansible 5.x. Reverting the ansible version back while keeping molecule on the latest version resolves the issue. At least, I don't see changes to Molecule in the Pipfile.lock but I may not recognize a sub-dependency conflict or something. The following is the PR I was working on and has the exact Pipfile.lock changes that lead to this molecule error.

chore: update dependencies and move to ansible FQCNs artis3n/ansible-role-tailscale#165

Note after reverting my Pipfile.lock to the version on main and running molecule test, everything worked fine. Including when I re-ran pipenv update --dev; pipenv run molecule test. I assume because the symlink was persisted on the filesystem. After committing the code and rebuilding a Codespace environment, molecule again failed with this error with the latest version of ansible / ansible-core. I do see ansible-compat having updated here in my Pipfile.lock which would correlate with @sdarwin's observation.

From a Codespace:

WORKING:

@artis3n ➜ /workspaces/ansible-role-tailscale (main) $ pipenv run ansible --version
ansible [core 2.11.6] 
  config file = None
  configured module search path = ['/home/vscode/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/vscode/.local/share/virtualenvs/ansible-role-tailscale-tV5cXObL/lib/python3.10/site-packages/ansible
  ansible collection location = /home/vscode/.ansible/collections:/usr/share/ansible/collections
  executable location = /home/vscode/.local/share/virtualenvs/ansible-role-tailscale-tV5cXObL/bin/ansible
  python version = 3.10.0 (default, Nov 17 2021, 15:26:39) [GCC 10.2.1 20210110]
  jinja version = 3.0.3
  libyaml = True

@artis3n ➜ /workspaces/ansible-role-tailscale (main) $ pipenv run molecule --version
molecule 3.5.2 using python 3.10 
    ansible:2.11.6
    delegated:3.5.2 from molecule
    docker:1.1.0 from molecule_docker requiring collections: community.docker>=1.9.1

BROKEN:

@artis3n ➜ /workspaces/ansible-role-tailscale (ansible5.x) $ pipenv run ansible --version
ansible [core 2.12.1]
  config file = None
  configured module search path = ['/home/vscode/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/vscode/.local/share/virtualenvs/ansible-role-tailscale-tV5cXObL/lib/python3.10/site-packages/ansible
  ansible collection location = /home/vscode/.ansible/collections:/usr/share/ansible/collections
  executable location = /home/vscode/.local/share/virtualenvs/ansible-role-tailscale-tV5cXObL/bin/ansible
  python version = 3.10.0 (default, Nov 17 2021, 15:26:39) [GCC 10.2.1 20210110]
  jinja version = 3.0.3
  libyaml = True

@artis3n ➜ /workspaces/ansible-role-tailscale (ansible5.x) $ pipenv run molecule --version
molecule 3.5.2 using python 3.10 
    ansible:2.12.1
    delegated:3.5.2 from molecule
    docker:1.1.0 from molecule_docker requiring collections: community.docker>=1.9.1

scruffaluff · 2022-01-21T16:54:14Z

The issue is caused by the changes to the ansible_compat.runtime.Runtime.prepare_environment method that molecule calls at https://github.com/ansible-community/molecule/blob/a8d96aa86199e73347013aba7752d5dd73f68ed3/src/molecule/command/base.py#L111.

In version 1.0.0, ansible-compat added a parameter to the method, https://github.com/ansible-community/ansible-compat/blob/v1.0.0/src/ansible_compat/runtime.py#L299, called install_local. The parameter is set to False by default, which skips the installation of the collection or role.

Changing the invocation from scenario.config.runtime.prepare_environment() to scenario.config.runtime.prepare_environment(install_local=True) should fix the issue.

If you want, I can test the change and make a pull request.

ssbarnea · 2022-01-21T17:15:18Z

Please do. We do have a good test coverage so we should get an idea about the impact of this move.

bonddim · 2022-02-23T08:50:55Z

This issue is not fully related to merged fix.

The main reason is in that molecule doesn't look at the env variables exposed by ansible-compat during prerun.
molecule uses ansible-compat runtime in isolated mode - compat installs roles and collections to own .cache directory, but molecule doesn't know anything about it.

Here is the example of 'ANSIBLE_COLLECTIONS_PATH' and 'ANSIBLE_ROLES_PATH' variables:

INFO     Performing prerun...
INFO     Set ANSIBLE_LIBRARY=/home/runner/.cache/ansible-compat/d035ce/modules:/home/runner/.ansible/plugins/modules:/usr/share/ansible/plugins/modules
INFO     Set ANSIBLE_COLLECTIONS_PATH=/home/runner/.cache/ansible-compat/d035ce/collections:/home/runner/.ansible/collections:/usr/share/ansible/collections
INFO     Set ANSIBLE_ROLES_PATH=/home/runner/.cache/ansible-compat/d035ce/roles:/home/runner/.ansible/roles:/usr/share/ansible/roles:/etc/ansible/roles
INFO     Running default > dependency
Molecule default > dependency
  INFO     Running ansible-galaxy collection install -v community.general:>=4.1.0
  WARNING  Skipping, missing the requirements file.
  WARNING  Skipping, missing the requirements file.
INFO     Running default > lint
Molecule default > lint
INFO     Running default > cleanup
Molecule default > cleanup
INFO     Running default > destroy
Molecule default > destroy
  DEBUG: ANSIBLE ENVIRONMENT:
  ANSIBLE_COLLECTIONS_PATH: /home/runner/.cache/molecule/test_command_init_scenario0/default/collections:/home/runner/.ansible/collections:/usr/share/ansible/collections:/etc/ansible/collections
  ANSIBLE_ROLES_PATH: '/home/runner/.cache/molecule/test_command_init_scenario0/default/roles:/tmp/pytest-of-runner/pytest-3:/home/runner/.ansible/roles:/usr/share/ansible/roles:/etc/ansible/roles:'

#3516) * Fix wrong environment variables being passed to ansible compat (#3404) * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

giner · 2022-04-27T04:50:02Z

1136ea5 (PR #3516) seems to break local install again:

ERROR! the role 'myrole' was not found in /home/myuser/myansible/roles/myrole/molecule/default/roles:/home/myuser/.cache/ansible-compat/e1c2a0/roles:/home/myuser/.ansible/roles:/usr/share/ansible/roles:/etc/ansible/roles:/home/myuser/myansible/roles/myrole/molecule/default

The error appears to be in '/home/myuser/myansible/roles/myrole/molecule/default/converge.yml': line 15, column 15, but may
be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:

      include_role:
        name: "myrole"
              ^ here

zhan9san · 2022-04-27T06:52:25Z

Hi @bonddim

The main reason is in that molecule doesn't look at the env variables exposed by ansible-compat during prerun.
molecule uses ansible-compat runtime in isolated mode - compat installs roles and collections to own .cache directory, but molecule doesn't know anything about it.

Could you help verify whether #3514 and ansible/ansible-compat#135 address your issue?

zhan9san · 2022-04-27T07:15:05Z

@bonddim

compat installs roles and collections to own .cache directory,

Currently, ansible-compat didn't install the roles or collections into .cache directory but self.config.default_roles_path[0]), even the install_local is hard-code True, See below

https://github.com/ansible/ansible-compat/blob/05d94d4d1b484b05c53c9b5844e4187d9872c258/src/ansible_compat/runtime.py#L487

As we all know, ansible-compat would create an isolated environment, https://github.com/ansible-community/molecule/blob/d62b88e203d55480f2a262dbbe287b59ddea3256/src/molecule/app.py#L10 and then, a cache_dir will be created. But unfortunately, this cache_dir is not used.

Instead, it is in this ansible/ansible-compat#135 that roles or collections would be installed to cache_dir. See below

https://github.com/ansible/ansible-compat/pull/135/files#diff-52de7db791930eca228ba958a8fe0d53d2695de26a1d6e1aa1906da28f607892R439-R445

bonddim · 2022-04-27T15:48:35Z

Hi @zhan9san

Could you help verify whether #3514 and ansible/ansible-compat#135 address your issue?

Those fixes issues with roles, but not collections.
I created simple check for this here.
As you can see, in test with both commits (toxenv all), role directory is symlinked to cache_dir and imported successfully.

zhan9san · 2022-04-28T09:17:44Z

Hi @bonddim

Those fixes issues with roles, but not collections.

Could you please give more detailed information about the collections?

I verified it through the following commands, and it works as expected

Set up a test environment

Ensure docker daemon is available, and then

pip install git+https://github.com/zhan9san/molecule@feature/add-cache-dir-into-ansible-roles-path
pip install git+https://github.com/zhan9san/ansible-compat@feature/install-role-in-cache-dir
pip install molecule-docker

Create a role

$ molecule init role acme.my_new_role --driver-name docker

Run test

$ cd my_new_role
$ molecule test

Verify roles installed

$ ll ~/.cache/ansible-compat/b4d3f4/roles
total 0
lrwxr-xr-x   1 jackzhang  TNSOFTWARE\Domain Users    32B Apr 28 16:16 acme.my_new_role -> /Users/jackzhang/tmp/my_new_role

Define requirements

The roles and collections in requirements.yml would be installed to the isolated cache dir as well, if it exists.

Create a requirements.yml and the content is as below.

$ cat requirements.yml
---
roles:
  # Install a role from Ansible Galaxy.
  - name: geerlingguy.java
    version: 1.9.6

collections:
  # Install a collection from Ansible Galaxy.
  - name: geerlingguy.php_roles
    version: 0.9.3
    source: https://galaxy.ansible.com

Verify the roles and collections in `requirements.yml`

$ molecule dependency
$ ll ~/.cache/ansible-compat/b4d3f4/roles ~/.cache/ansible-compat/b4d3f4/collections/ansible_collections
/Users/jackzhang/.cache/ansible-compat/b4d3f4/collections/ansible_collections:
total 0
drwxr-xr-x  3 jackzhang  TNSOFTWARE\Domain Users    96B Apr 28 16:51 geerlingguy

/Users/jackzhang/.cache/ansible-compat/b4d3f4/roles:
total 0
lrwxr-xr-x   1 jackzhang  TNSOFTWARE\Domain Users    32B Apr 28 16:16 acme.my_new_role -> /Users/jackzhang/tmp/my_new_role
drwxr-xr-x  12 jackzhang  TNSOFTWARE\Domain Users   384B Apr 28 16:50 geerlingguy.java

zhan9san · 2022-04-28T10:01:40Z

@bonddim

You are right.

The isolated collections path installed by ansible-compat is added in ANSIBLE_COLLECTIONS_PATH in a new commit.

https://github.com/ansible-community/molecule/pull/3514/files#diff-32abfdacc4eeea878460d3bf0ae452ab074ca485bab0b0a01b3bb07775ac002dR422-R426

It may work now.

bonddim · 2022-04-28T10:02:07Z

@zhan9san
My issue:
pre-run installs collection while running inside ansible_collections/namespace/collection_name with dependencies defined in galaxy.yml file, but required collections aren't available for molecule.

During roles test (as you describe), all is ok

bonddim · 2022-04-28T10:07:33Z

@zhan9san

https://github.com/ansible-community/molecule/pull/3514/files#diff-32abfdacc4eeea878460d3bf0ae452ab074ca485bab0b0a01b3bb07775ac002dR422-R426

this fixes the issue with collections

ansible/molecule#3404

See: ansible/molecule#3404

jonlaokan · 2022-05-31T10:06:12Z

I have the same issue while running Molecule inside a vm.
molecule: 3.6.1

The SET PATH does not set anything at the moment:

zhan9san · 2022-05-31T10:41:09Z

@jonlaokan

Can you verify whether #3514 address your issue?

pip install git+https://github.com/zhan9san/molecule@feature/add-cache-dir-into-ansible-roles-path
pip install ansible-compat==2.1.0

ssbarnea · 2022-06-01T10:07:48Z

#3569 should fix it.

- ansible/molecule#3404 - ansible/molecule#3569 - ansible/molecule#3514 - https://pypi.org/project/molecule/#history - remove workaround for broken collection support

sdarwin added the bug label Jan 18, 2022

artis3n mentioned this issue Jan 20, 2022

molecule doesn't find role with ansible-compat == 1.0.0 ansible/ansible-compat#114

Closed

scruffaluff mentioned this issue Jan 22, 2022

Fix prepare_environment call for role installs #3409

Merged

Nubly mentioned this issue Mar 10, 2022

Allow configuration of install_local bool in prerun #3464

Closed

zhan9san mentioned this issue May 4, 2022

Add more tests related to Ansible runtime path modifications #3529

Merged

artis3n mentioned this issue May 24, 2022

chore(deps-dev): bump ansible-lint from 6.1.0 to 6.2.1 artis3n/ansible-role-tailscale#208

Closed

yabusygin added a commit to yabusygin/ansible-role-gitlab that referenced this issue May 26, 2022

Add workaround for Molecule issue #3404

6c1f8a0

ansible/molecule#3404

yabusygin added a commit to yabusygin/ansible-role-gitlab that referenced this issue May 26, 2022

Add workaround for Molecule issue #3404

60cadc9

ansible/molecule#3404

yabusygin added a commit to yabusygin/ansible-role-docker that referenced this issue May 30, 2022

Add workaround for Molecule issue #3404

45464f7

See: ansible/molecule#3404

yabusygin mentioned this issue May 30, 2022

Upgrade Molecule to version 3.6 yabusygin/ansible-role-docker#78

Merged

ssbarnea closed this as completed Jun 1, 2022

0uep mentioned this issue Jun 14, 2022

ERROR! the role 'acme.myrole' was not found. "symlink to current repository" not used?! #3581

Closed

nodiscc added a commit to nodiscc/xsrv that referenced this issue Jun 15, 2022

tools: update molecule to 4.0.0

9cd823e

- ansible/molecule#3404 - ansible/molecule#3569 - ansible/molecule#3514 - https://pypi.org/project/molecule/#history - remove workaround for broken collection support

yabusygin mentioned this issue Aug 2, 2022

Update testing requirements yabusygin/ansible-role-docker#87

Merged

nodiscc added a commit to nodiscc/xsrv that referenced this issue Aug 30, 2022

tools: update molecule to 4.0.0

258b0e7

- ansible/molecule#3404 - ansible/molecule#3569 - ansible/molecule#3514 - https://pypi.org/project/molecule/#history - remove workaround for broken collection support

dalepotter mentioned this issue Jan 8, 2023

Role not found when using custom ANSIBLE_ROLES_PATH #3806

Open

pwalczysko mentioned this issue Jun 5, 2023

Try fixing build ome/ansible-role-omero-web#45

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ANSIBLE_ROLES_PATH not set correctly? #3404

ANSIBLE_ROLES_PATH not set correctly? #3404

sdarwin commented Jan 18, 2022 •

edited

Loading

ssbarnea commented Jan 19, 2022

sdarwin commented Jan 19, 2022 •

edited

Loading

sdarwin commented Jan 19, 2022

ssbarnea commented Jan 19, 2022

sdarwin commented Jan 19, 2022

artis3n commented Jan 20, 2022 •

edited

Loading

scruffaluff commented Jan 21, 2022

ssbarnea commented Jan 21, 2022

bonddim commented Feb 23, 2022 •

edited

Loading

giner commented Apr 27, 2022

zhan9san commented Apr 27, 2022

zhan9san commented Apr 27, 2022

bonddim commented Apr 27, 2022

zhan9san commented Apr 28, 2022

zhan9san commented Apr 28, 2022

bonddim commented Apr 28, 2022

bonddim commented Apr 28, 2022

jonlaokan commented May 31, 2022

zhan9san commented May 31, 2022

ssbarnea commented Jun 1, 2022

ANSIBLE_ROLES_PATH not set correctly? #3404

ANSIBLE_ROLES_PATH not set correctly? #3404

Comments

sdarwin commented Jan 18, 2022 • edited Loading

Issue Type

Molecule and Ansible details

Desired Behavior

Actual Behaviour

ssbarnea commented Jan 19, 2022

sdarwin commented Jan 19, 2022 • edited Loading

sdarwin commented Jan 19, 2022

ssbarnea commented Jan 19, 2022

sdarwin commented Jan 19, 2022

artis3n commented Jan 20, 2022 • edited Loading

scruffaluff commented Jan 21, 2022

ssbarnea commented Jan 21, 2022

bonddim commented Feb 23, 2022 • edited Loading

giner commented Apr 27, 2022

zhan9san commented Apr 27, 2022

zhan9san commented Apr 27, 2022

bonddim commented Apr 27, 2022

zhan9san commented Apr 28, 2022

Set up a test environment

Create a role

Run test

Verify roles installed

Define requirements

Verify the roles and collections in requirements.yml

zhan9san commented Apr 28, 2022

bonddim commented Apr 28, 2022

bonddim commented Apr 28, 2022

jonlaokan commented May 31, 2022

zhan9san commented May 31, 2022

ssbarnea commented Jun 1, 2022

sdarwin commented Jan 18, 2022 •

edited

Loading

sdarwin commented Jan 19, 2022 •

edited

Loading

artis3n commented Jan 20, 2022 •

edited

Loading

bonddim commented Feb 23, 2022 •

edited

Loading

Verify the roles and collections in `requirements.yml`