0.20.0rc1 can't pip install on WSL without GCC

[joshpeng]

Describe the bug

When using Windows Subsystem for Linux, the latest 0.20.0rc1 release can't install into a new Python environment due to the tree-sitter package which seems to require GCC. This was not a requirement for 0.19.1 or 0.20.0b1.

Steps To Reproduce

conda create -n dbt_020rc1 python=3.8
conda activate dbt_020rc1
pip install dbt-bigquery==0.20.0rc1

It errors out with

  error: command 'gcc' failed with exit status 1
  ----------------------------------------
  ERROR: Failed building wheel for tree-sitter
Failed to build tree-sitter
ERROR: Could not build wheels for tree-sitter which use PEP 517 and cannot be installed directly

Expected behavior

Would have expected this to work without GCC like in 0.19.1 and 0.20.0b1. Installing GCC via sudo apt install gcc makes it work again though. Perhaps just adding this tidbit of info in an upgrade guide would suffice.

[jtcohen6]

Thanks for opening @joshpeng, and for your patience in getting the installation working! You're right that tree-sitter and tree-sitter-jinja2 are new dependencies in 0.20.0rc1:

https://github.com/fishtown-analytics/dbt/blob/abe8e839458d0288f76cb6950e79c77e3b1627cc/core/setup.py#L76-L77

And from the py-tree-sitter docs:

This package currently only works with Python 3. There are no library dependencies, but you do need to have a C compiler installed.

I'm inclined toward your recommendation here, in the hopes that installing GCC / a C compiler is not too heavy of a lift:

Perhaps just adding this tidbit of info in an upgrade guide would suffice.

If it proves to be a significant source of complexity, we could move tree-sitter and tree-sitter-jinja2 into an extra.

(cc @kwigley @nathaniel-may: I'd be curious to hear your thoughts)

[aa-matthias]

I'm having an issue installing 0.20.0rc1 on windows - also related to tree-sitter where pipenv install errors out with:

[pipenv.exceptions.InstallError]: Collecting tree-sitter==0.19.0
[pipenv.exceptions.InstallError]:   Using cached tree_sitter-0.19.0.tar.gz (112 kB)
[pipenv.exceptions.InstallError]: ERROR: Disabling PEP 517 processing is invalid: project specifies a build backend of setuptools.build_meta in pyproject.toml
ERROR: Couldn't install package: tree-sitter

the root cause for this had been that i've not installed Visual Studio 2019 Build Tools on my local machine so building tree_sitter failed. In that case pipenv tries to install a second time with --no-pep-517.

i now managed to install the build tools (2+ gb) so this comes on a heavy premium and additional steps as long as no pip wheel is provided for tree_sitter.

[kwigley]

@joshpeng @aa-matthias thanks for calling this out. There is no way to provide wheels distribution for the 3rd party tree-sitter dependency. I think this is a good enough reason to make this dependency opt-in as an "extra" dependency as this will likely break a lot of upgrades and first-time installs.

Along with a note in docs that this method requires a C compiler, installing an extra dependency via pip will look something like:

pip install dbt[tree-sitter]

[jtcohen6]

We ended up resolving this by rewriting the "experimental parser" (code leveraging tree-sitter) in Rust. That gets us better type checking and, crucially for this issue, the ability to package up the tree-sitter binary in our wheels.

We're going to cut 0.20.0rc2 tomorrow, and it should be installable without needing a C compiler.

In the meantime, I'm going to close this issue as having been resolved by #3497