Jinja2 3.1 Breaks filter #77413

rgajason · 2022-03-30T14:00:09Z

Summary

Version 3.1 of Jinja2 removed deprecated code, including environmentfilter: pallets/jinja#1544

Attempting to execute Ansible operations that use filter with Jinja2 >= 3.1 results in:

[WARNING]: Skipping plugin (/opt/python3/lib/python3.10/site-packages/ansible/plugins/filter/core.py) as it seems to be invalid: cannot import name 'environmentfilter' from 'jinja2.filters'  (/opt/python3/lib/python3.10/site-packages/jinja2/filters.py)

This issue appears to have been originally addressed here: #74667

Issue Type

Bug Report

Component Name

filter

Ansible Version

$ ansible --version
ansible [core 2.12.3]
  config file = None
  configured module search path = ['/home/jenkins/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /opt/python3/lib/python3.10/site-packages/ansible
  ansible collection location = /home/jenkins/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/local/bin/ansible
  python version = 3.10.4 (main, Mar 24 2022, 16:14:08) [GCC 4.8.5 20150623 (Red Hat 4.8.5-44)]
  jinja version = 3.1.1
  libyaml = False

Configuration

$ ansible-config dump --only-changed
<no output>

OS / Environment

Linux (EL7)
Python 3.10.4 compiled from source
Jinja2 3.1.1
Ansible 5.5.0 (according to pip3 freeze)
Ansible Core 2.12.3 (according to pip3 freeze)

Same results with Ansible 2.10.7 and Ansible Base 2.10.9 using Python 3.9.10.

Steps to Reproduce

Install Jinja2 version 3.1.x
Execute playbook

Expected Results

No Python import error

Actual Results

[WARNING]: Skipping plugin (/opt/python3/lib/python3.10/site-packages/ansible/plugins/filter/core.py) as it seems to be invalid: cannot import name 'environmentfilter' from 'jinja2.filters'  (/opt/python3/lib/python3.10/site-packages/jinja2/filters.py)

Code of Conduct

I agree to follow the Ansible Code of Conduct

The text was updated successfully, but these errors were encountered:

ansibot · 2022-03-30T14:06:03Z

Files identified in the description:

test/lib/ansible_test/_internal/cli/commands/coverage/analyze/targets/filter.py

If these files are incorrect, please update the component name section of the description or use the !component bot command.

click here for bot help

bcoca · 2022-03-30T14:08:43Z

fixed in #75881, which applied it to 2.13

mkrizek · 2022-03-30T14:44:34Z

This actually looks more like a duplicate of #77406, #77383 and #77356.

Same results with Ansible 2.10.7 and Ansible Base 2.10.9 using Python 3.9.10.

This is expected, the versions of Ansible that this was fixed (#74666) in are 2.9.22, 2.10.10, 2.11.1 and 2.12.0.

Ansible Core 2.12.3 (according to pip3 freeze)

This should not be the case. Can you double check that you are indeed running 2.12.3 version and that /opt/python3/lib/python3.10/site-packages/ansible/__init__.py (inferred from your output) contains the fix from f99d024#diff-8dc7a6a3bbcf9fbd56c4f0a97c98cf20c58ad49c588a1e6937c4197c24deed0f? Otherwise it might indicate that something is wrong with the installation.

needs_info

xfyre · 2022-03-31T01:42:18Z

@mkrizek Could you confirm it was fixed in 2.9.22?

I've upgraded to this version and I'm still getting the same issue:

2022-03-31 01:39:05
dcnet-build
TASK [Retrieve project names] **************************************************
2022-03-31 01:39:05
dcnet-build
[WARNING]: Skipping plugin (/usr/local/lib/python3.9/dist-
2022-03-31 01:39:05
dcnet-build
packages/ansible/plugins/filter/core.py) as it seems to be invalid: cannot
2022-03-31 01:39:05
dcnet-build
import name 'environmentfilter' from 'jinja2.filters'
2022-03-31 01:39:05
dcnet-build
(/usr/local/lib/python3.9/dist-packages/jinja2/filters.py)
2022-03-31 01:39:05
dcnet-build
[WARNING]: Skipping plugin (/usr/local/lib/python3.9/dist-
2022-03-31 01:39:05
dcnet-build
packages/ansible/plugins/filter/mathstuff.py) as it seems to be invalid: cannot
2022-03-31 01:39:05
dcnet-build
import name 'environmentfilter' from 'jinja2.filters'
2022-03-31 01:39:05
dcnet-build
(/usr/local/lib/python3.9/dist-packages/jinja2/filters.py)
2022-03-31 01:39:05
dcnet-build
fatal: [localhost]: FAILED! => {"msg": "template error while templating string: No filter named 'flatten'.. String: {{ commit_message | regex_findall('(dcnet-server|dcnet-nuxt|dcnet-redir)@([a-f0-9]+)') | flatten }}"}
2022-03-31 01:39:05
dcnet-build

2022-03-31 01:39:05
dcnet-build
PLAY RECAP *********************************************************************
2022-03-31 01:39:05
dcnet-build
localhost                  : ok=0    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0   
2022-03-31 01:39:05
dcnet-build

Akasurde · 2022-03-31T04:49:38Z

@xfyre For me it works -

---
- hosts: localhost
  gather_facts: no
  tasks:
    - debug:
        msg: "{{ 1 | pow(2) }}"

PLAY [localhost] ***************************************************************

TASK [debug] *******************************************************************
ok: [localhost] => {
    "msg": "1.0"
}

PLAY RECAP *********************************************************************
localhost                  : ok=1    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

ansible --version
ansible 2.9.22
  config file = None
  configured module search path = ['/Users/akasurde/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /Volumes/data/src/ansible/lib/ansible
  executable location = /Volumes/data/src/ansible/bin/ansible
  python version = 3.9.10 (main, Jan 15 2022, 11:48:00) [Clang 13.0.0 (clang-1300.0.29.3)]

mkrizek · 2022-03-31T06:57:15Z

@mkrizek Could you confirm it was fixed in 2.9.22?

Yes.

I've upgraded to this version and I'm still getting the same issue:

2022-03-31 01:39:05
dcnet-build
TASK [Retrieve project names] **************************************************
2022-03-31 01:39:05
dcnet-build
[WARNING]: Skipping plugin (/usr/local/lib/python3.9/dist-
2022-03-31 01:39:05
dcnet-build
packages/ansible/plugins/filter/core.py) as it seems to be invalid: cannot
2022-03-31 01:39:05
dcnet-build
import name 'environmentfilter' from 'jinja2.filters'
2022-03-31 01:39:05
dcnet-build
(/usr/local/lib/python3.9/dist-packages/jinja2/filters.py)
2022-03-31 01:39:05
dcnet-build
[WARNING]: Skipping plugin (/usr/local/lib/python3.9/dist-
2022-03-31 01:39:05
dcnet-build
packages/ansible/plugins/filter/mathstuff.py) as it seems to be invalid: cannot
2022-03-31 01:39:05
dcnet-build
import name 'environmentfilter' from 'jinja2.filters'
2022-03-31 01:39:05
dcnet-build
(/usr/local/lib/python3.9/dist-packages/jinja2/filters.py)
2022-03-31 01:39:05
dcnet-build
fatal: [localhost]: FAILED! => {"msg": "template error while templating string: No filter named 'flatten'.. String: {{ commit_message | regex_findall('(dcnet-server|dcnet-nuxt|dcnet-redir)@([a-f0-9]+)') | flatten }}"}
2022-03-31 01:39:05
dcnet-build

2022-03-31 01:39:05
dcnet-build
PLAY RECAP *********************************************************************
2022-03-31 01:39:05
dcnet-build
localhost                  : ok=0    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0   
2022-03-31 01:39:05
dcnet-build

Assuming the location from the provided output, what is the output of the following command?

grep "patch Jinja2 >= 3.0 for backwards compatibility" /usr/local/lib/python3.9/dist-packages/ansible/__init__.py

xfyre · 2022-03-31T20:58:16Z

@mkrizek Thanks for looking into this. I appears that, indeed, everything is working is expected, and my problem was related to the CodeShip "cache reset" feature not doing what it should (cached docker image with an old version somehow stuck and kept giving me the same error).

My builds today are working with 2.9.22.

rgajason · 2022-04-01T01:52:43Z

Things are also working for me. The Docker image I was testing that originally led to this bug report was destroyed and I don't have a list of Python modules/versions that were installed outside of ansible/ansible-base/Jinja2. Several modules that I install have changed their versions and dependency requirements since I reported this and I'm unable to find the right combination to recreate.

Thanks to all who spent time on this.

mkrizek · 2022-04-01T06:13:29Z

Closing as per above.

If you have further questions please stop by IRC or the mailing list:

IRC: #ansible on irc.libera.chat
mailing list: https://groups.google.com/forum/#!forum/ansible-project

rgajason mentioned this issue Mar 30, 2022

Jinja2 3.1 Breaks filter ansible-collections/ansible.utils#153

Closed

ansibot added affects_2.12 bug This issue/PR relates to a bug. needs_triage Needs a first human triage before being processed. support:core This issue/PR relates to code supported by the Ansible Engineering Team. test This PR relates to tests. labels Mar 30, 2022

ansibot added the needs_info This issue requires further information. Please answer any outstanding questions. label Mar 30, 2022

jborean93 removed the needs_triage Needs a first human triage before being processed. label Mar 31, 2022

ansibot removed the needs_info This issue requires further information. Please answer any outstanding questions. label Apr 1, 2022

mkrizek closed this as completed Apr 1, 2022

tormi mentioned this issue Apr 4, 2022

Jinja 2 imports have been deprecated wunderio/WunderMachina#369

Closed

ansible locked and limited conversation to collaborators Apr 8, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Jinja2 3.1 Breaks filter #77413

Jinja2 3.1 Breaks filter #77413

rgajason commented Mar 30, 2022 •

edited

ansibot commented Mar 30, 2022

bcoca commented Mar 30, 2022 •

edited

mkrizek commented Mar 30, 2022

xfyre commented Mar 31, 2022

Akasurde commented Mar 31, 2022

mkrizek commented Mar 31, 2022

xfyre commented Mar 31, 2022

rgajason commented Apr 1, 2022

mkrizek commented Apr 1, 2022

Jinja2 3.1 Breaks filter #77413

Jinja2 3.1 Breaks filter #77413

Comments

rgajason commented Mar 30, 2022 • edited

Summary

Issue Type

Component Name

Ansible Version

Configuration

OS / Environment

Steps to Reproduce

Expected Results

Actual Results

Code of Conduct

ansibot commented Mar 30, 2022

bcoca commented Mar 30, 2022 • edited

mkrizek commented Mar 30, 2022

xfyre commented Mar 31, 2022

Akasurde commented Mar 31, 2022

mkrizek commented Mar 31, 2022

xfyre commented Mar 31, 2022

rgajason commented Apr 1, 2022

mkrizek commented Apr 1, 2022

rgajason commented Mar 30, 2022 •

edited

bcoca commented Mar 30, 2022 •

edited