Correctly handle ANSIBLE_FILTER_PLUGINS for plugin vendored filter functions

[tumbl3w33d]

I've just started using this and was confused by some errors the generated default files emit.

The first one was as edge case that I, unfortunately, cannot reproduce anymore right now. It wouldn't hurt to handle it gracefully anyway, I guess

$ molecule test

# in destroy.yml

TASK [Populate instance config from file] **************************************
[WARNING]: Unable to find
'/home/myuser/.cache/molecule/myrole/default/instance_config.yml' in
expected paths (use -vvvvv to see paths)
fatal: [localhost]: FAILED! => {"msg": "An unhandled exception occurred while running the lookup plugin 'file'. Error was a <class 'ansible.errors.AnsibleError'>, original message: could not locate file in lookup: /home/myuser/.cache/molecule/myrole/default/instance_config.yml"}

TASK [Populate instance config when file missing] ******************************
ok: [localhost]

It's not a big deal after all, because this happens in a rescue block, but it confused me at first until I found that out. I'd suggest to add a failed_when that catches this expected condition. Something like:

    - name: Populate the instance config
      block:
        - name: Populate instance config from file
          set_fact:
            instance_conf: "{{ lookup('file', molecule_instance_config) | from_yaml }}"
            skip_instances: false
          register: instance_config_lookup
          failed_when: instance_config_lookup is not success and 'could not locate file in lookup' not in instance_config_lookup.stderr

---

A second thing that caused problems was the generated create.yml as it uses the filter molecule_get_hetznercloud_networks that are not in the filter_path and there's no explanation in the documentation how to fix that.

TASK [Create private network(s)] ***********************************************
fatal: [localhost]: FAILED! => {"msg": "template error while templating string: No filter named 'molecule_get_hetznercloud_networks'.. String: {{ molecule_yml.platforms|molecule_get_hetznercloud_networks('networks') }}"}

I could try pointing that to the virtualenv path but it's not really stable as the python version changes. When you rename/delete the create.yml then the "implicit" one is executed and it finds its filter. Copying this filter to the role does not seem like a maintainable solution to me either.

[decentral1se]

Hi @tumbl3w33d! Thanks for taking some time on this one.

I'd suggest to add a failed_when that catches this expected condition.

💯 will happily merge a PR for this 💯

A second thing that caused problems was the generated create.yml as it uses the filter molecule_get_hetznercloud_networks that are not in the filter_path and there's no explanation in the documentation how to fix that.

This sounds like a bug. @ggggut do you have any idea why that might be happening (from #29). I'll note that this doesn't happen in the test suites so I am wondering if this is something in your local setup @tumbl3w33d but I can't be quite sure what. I don't have much time this week but I'll try to look into this shortly. If you have any fix suggestions, let me know!

[tumbl3w33d]

Thank you for your fast reply. I'll try to gather some more in-depth information about what causes the filter path issue.

[tumbl3w33d]

I reproduced it in a fresh folder with a fresh role:

# created a new directory 'foo' and stepped into it
# then we create a virtualenv
[myuser@myworkstation foo]$ python3 -m venv venv
# and activate it
[myuser@myworkstation foo]$ . venv/bin/activate
# we install the modules ansible, molecule[hetznercloud] and molecule-hetznercloud
(venv) [myuser@myworkstation foo]$ pip install -r requirements.txt
[…]
# we create a fresh role
(venv) [myuser@myworkstation foo]$ molecule init role foo -d hetznercloud
INFO     Initializing new role foo...
No config file found; using defaults
- Role foo was created successfully
INFO     Initialized role in /home/myuser/projects/foo/foo successfully.

Then I prepared the run:

• set HCLOUD_TOKEN via direnv file
• disable linting in molecule.yml because it's irrelevant here and keeps from reaching the right playbook
• step into the role folder with cd foo
• molecule --debug test

  TASK [Create private network(s)] ***********************************************
fatal: [localhost]: FAILED! => {"msg": "template error while templating string: No filter named 'molecule_get_hetznercloud_networks'.. String: {{ molecule_yml.platforms|molecule_get_hetznercloud_networks('networks') }}"}

• debug output: ANSIBLE_FILTER_PLUGINS=/home/myuser/projects/foo/venv/lib/python3.9/site-packages/molecule/provisioner/ansible/plugins/filter:/home/myuser/.cache/molecule/foo/default/plugins/filter:/home/myuser/projects/foo/foo/plugins/filter:/home/myuser/.ansible/plugins/filter:/usr/share/ansible/plugins/filter
• actual path: /home/myuser/projects/foo/venv/lib/python3.9/site-packages/molecule_hetznercloud/playbooks/filter_plugins/get_hetznercloud_networks.py

Not sure yet if I did something wrong at some point, but the path of the virtualenv is different from where molecule is searching.

[ekeih]

Hello, coworker of @ggggut here. I can reproduce this in a virtualenv.
I don't 100% know how molecule drivers are supposed to work, mostly it is just guessing and looking at existing drivers. But after looking more at other drivers, especially https://github.com/ansible-community/molecule-docker (which also uses a filter plugin), I think the create.yml and destroy.yml playbooks are not supposed to be in the cookiecutter template. This way they are not templated during the role creation and instead the original playbooks of the driver are used, which are able to find the filter plugin at the correct location.
I tested this solutions locally and everything seems to work just fine. @decentral1se do you think this is the proper solution or am I missing something that requires the files to be templated as part of the role creation? If you give your okay I can provide a pull request.

[decentral1se]

Hey @ekeih, thanks for looking into this! So, we need to pass our own create.yml because we have specific Hetzner Cloud logic we need to take care of (like creating networks). You can see an example of another drive which is doing this in https://github.com/ansible-community/molecule-vmware/tree/master/molecule_vmware/cookiecutter/%7B%7Bcookiecutter.molecule_directory%7D%7D/%7B%7Bcookiecutter.scenario_name%7D%7D.

From the good investigation of @tumbl3w33d it looks like we instead need to hook into the ANSIBLE_FILTER_PLUGINS logic to make sure the correct path is available. Back in upstream Molecule, that is defined like so:

https://github.com/ansible-community/molecule/blob/3d3ce253f396ac0117659bf9f2cbc06c7377b508/src/molecule/provisioner/ansible.py#L461

We can hook into that in our own subclass here in this plugin over at:

molecule-hetznercloud/molecule_hetznercloud/driver.py

Line 10 in ef8c7e3

class HetznerCloud(Driver):

We need to get something like /home/myuser/projects/foo/venv/lib/python3.9/site-packages/molecule_hetznercloud/playbooks/filter_plugins/ onto this path while preserving the existing paths?

Does anyone have energy/time for a PR for this?

[tumbl3w33d]

I saw that an active virtualenv comes with an env var that we can rely on to build the path:

VIRTUAL_ENV=/home/myuser/projects/somepath/myproject/venv

Not sure how to handle the moving python version, though.

Edit: $VIRTUAL_ENV/lib/*/site-packages/molecule_hetznercloud/playbooks/filter_plugins using glob magic would do the trick. Might need to handle multiple python folders appearing there and add all of them

[ekeih]

Thanks for the additional insights @decentral1se :)

I understand that we need our own create.yml, but I think we don't need it in the driver in .../site-packages/molecule_hetznercloud/... and in the templated role we get by running molecule init role foo -d hetznercloud. If I understand the code correctly there is nothing special templated by cookiecutter into the file, it is just a raw copy of the same code that is also installed with the driver.

I created #33 to better show what I mean. If I use the code from this branch everything works as expected. Maybe you can try the branch and check if it also works for you or if I am missing a scenario where the templated copy of the playbooks is required.