[CT-255] [Bug] ImportError: cannot import name soft_unicode from markupsafe

[sumit-sampang-rai]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

Markup recently released its new version 2.1.0 which is essential for Jinja2 package. Jinja has already released version 3.x.x and dbt still uses version 2.x.x and their developers won't support 2.x.x for some time. Now the command dbt deps throws an error.

Expected Behavior

While running dbt deps the packages should install instead of throwing errors.

Steps To Reproduce

No response

Relevant log output

Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.8.12/x64/bin/dbt", line 5, in <module>
    from dbt.main import main
  File "/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/dbt/main.py", line 11, in <module>
    import dbt.version
  File "/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/dbt/version.py", line 11, in <module>
    import dbt.semver
  File "/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/dbt/semver.py", line 8, in <module>
    import dbt.utils
  File "/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/dbt/utils.py", line 9, in <module>
    import jinja2
  File "/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/jinja2/__init__.py", line 12, in <module>
    from .environment import Environment
  File "/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/jinja2/environment.py", line 25, in <module>
    from .defaults import BLOCK_END_STRING
  File "/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/jinja2/defaults.py", line 3, in <module>
    from .filters import FILTERS as DEFAULT_FILTERS  # noqa: F401
  File "/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/jinja2/filters.py", line 13, in <module>
    from markupsafe import soft_unicode
ImportError: cannot import name 'soft_unicode' from 'markupsafe' (/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/markupsafe/__init__.py)
Error: Process completed with exit code 1.

Environment

- OS: MacOS Monterey 12.2
- Python: 3.9.6
- dbt: 1.0.1

What database are you using dbt with?

snowflake

Additional Context

No response

[sumit-sampang-rai]

If someone is facing this error and need an urgent fix for your CI/CD or job in production, you can downgrade your MarkupSafe into version 2.0.1 manually. If you installed dbt-core with pip then you can run pip3 install --force-reinstall MarkupSafe==2.0.1

[abhinav-kumar-17]

If building from source(pip install .), the Jinja2 version can be bumped to 3.0.3 as well in the .core/setup.py. This does include some breaking changes in Jinja2

[scottpaulin]

Steps to reproduce

Building this Dockerfile reproduces the error

FROM ubuntu:20.04
# Pip e.t.c.
RUN apt-get update -y && apt-get install -y python3-pip python3-dev build-essential

# Install system requirements https://docs.getdbt.com/dbt-cli/install/pip#ubuntudebian
RUN apt-get install -y git libpq-dev python-dev python3-pip
RUN apt-get remove python-cffi
RUN pip install --upgrade cffi
RUN pip install cryptography~=3.4

# RUN pip install MarkupSafe==2.0.1

RUN pip install dbt-core

# python 3.8
RUN python3 --version

# Offending line
RUN dbt --version

ENTRYPOINT ["echo", "Hello, World"]

[yu-iskw]

MEMO: pallets/jinja#1587

[ftupas]

If someone is facing this error and need an urgent fix for your CI/CD or job in production, you can downgrade your MarkupSafe into version 2.0.1 manually. If you installed dbt-core with pip then you can run pip3 install --force-reinstall MarkupSafe==2.0.1

Thanks, had this issue in our CI/CD and it worked liked a charm!

[jtcohen6]

Thanks for opening @mailtosumitrai, and for weighing in everyone! I think our approach here will need to look like:

• Pinning MarkupSafe==2.0.1 in our production environments, and in the next patch release of dbt (v1.0.2, currently available as a release candidate)
• Looking into whether we can safely upgrade to Jinja3 in an upcoming minor version. We've hesitated to do this, given the risk of breaking changes to user/project code—but we need to weigh that against the risk of running into this type of dependency/installation issue, given that Jinja2 is no longer officially supported and won't be patched accordingly.