[CT-650] [Bug] python 3.7 version incompatible with networkx

[jpmmcneill]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

Networkx is pinned across several minor versions at the moment

dbt-core/core/setup.py

Line 64 in 72c17c4

"networkx>=2.3,<3",

Networkx minor versions (or maybe even patches) can introduce python version breaking changes for DBT's implementation. For example 2.3 was for python 3.5, 3.6, 3.7, while 2.8 is listed as being >3.8.

The current version of networkx (2.8) worked fine with 3.7 for DBT until a recent release
https://github.com/networkx/networkx/releases/tag/networkx-2.8.1rc1

this prerelease seems to be installed at the moment. This introduced use of functools.cached_property in networkx.graph which has made many versions of DBT require python 3.8, despite that not being clear.

For example, we are using 1.0.1 which is now unstable for python 3.7 users.

Expected Behavior

I'd expect this to be more stringently pinned to ensure that old versions of DBT aren't subject to versions changing in other modules.

Steps To Reproduce

No response

Relevant log output

No response

Environment

- OS: Mac
- Python: 3.7
- dbt: 1.0.1, true for everything until 1.1.0 AFAIK

What database are you using dbt with?

snowflake

Additional Context

No response

[jpmmcneill]

The interesting thing is that this is only because of the release candidate being used on the networkx side. networkx 2.8 (despite claiming it's 3.8 only) works for 3.7 in the context of DBT

[jtcohen6]

Good call! Tagging in Team:Execution since networkx is most relevant to their work.

I'm not sure why a prerelease version of networkx would be installed when installing dbt-core...? Also, if py37 had been retained for 2.8.0 — 2.8.1 is patch, not a new minor version :(

In any case, it sounds like we need to update the dependency conditionally based on the Python version. I don't know the perfect setup.py incantation for this, but something like:

 "networkx>=2.3,<2.8.1"; python_version=="3.7"',
 "networkx>=2.3,<3"; python_version>"3.7"',

And then backporting that change to 1.1.latest + 1.0.latest. This does fall under the purview of "critical support" IMO, since it prevents people from being able to install and use v1.0 with an officially supported version of Python.

[jpmmcneill]

Yeah - I have no clue why that new version of networkx snuck in - sorry that I can't be more helpful here.

Thankfully for us it was relatively painless to bump:

python -> 3.8
dbt -> 1.1

I doubt that everyone will have such a nice time 🐍
Critical support would be much appreciated