[Feature] Use Poetry for development

[NiallRees]

Is there an existing feature request for this?

• I have searched the existing issues

Describe the Feature

Poetry makes developing projects extremely easy, making getting started as simple as:

poetry install
poetry shell

Describe alternatives you've considered

No response

Who will this benefit?

No response

Are you interested in contributing this feature?

No response

Anything else?

No response

[jtcohen6]

Thanks for opening @NiallRees, and so sorry for the long delay in getting back to you!

I haven't (yet) tried poetry, but broadly it reminds me of some thinking we did around pipx a year ago: #3022

From perusing the docs, I sense that this change would/could involve:

• Replacing setup.py with pyproject.toml
• Updating CONTRIBUTING.md + Makefile
• Our release process, which is (at least) much more clearly laid out than it used to be

Do you see the biggest benefit here as making it easier for external contributors to install dbt-core from source locally? That's a barrier I'm always interested in lowering

[NiallRees]

Do you see the biggest benefit here as making it easier for external contributors to install dbt-core from source locally? That's a barrier I'm always interested in lowering

Yes absolutely. Poetry handles the creation of the development environment (no more venv invocation), and will create a poetry.lock file for a deterministic set of dependencies. There's also a bunch of other stuff which is over my head in relation to the pyproject.toml and PEP 518.

We should very soon see editable installations too once a new release is cut: python-poetry/poetry#5056

[leahwicz]

@iknox-fa have you ever used this with Python before? Just curious if you have any background on this

[kwigley]

I attempted to do this with Core about a year ago. A blocker I ran into was due to the monorepo setup that dbt had at the time (dbt-core + dbt-postgres + dbt-snowflake + dbt-redshift + dbt-bigquery) and editable installing these packages for development. I wasn't able to figure out how to install local packages in editable mode when setting up a local development environment and require the same packages from PyPI when publishing the packages without editing the pyproject.toml. This is probably not an issue anymore because:

1. the dbt package does not exist anymore
2. we split out the database adapters into separate repos

and, even if we still run into this issue, this may help! https://github.com/jacksmith15/poetry-workspace-plugin

I'm very much in favor of this! vitualenvs are.. not fun to manage

[iknox-fa]

@iknox-fa have you ever used this with Python before? Just curious if you have any background on this

I have! Poetry is nice, but in the long run I don't know that I'd ever advocate for enforcing it's use.

That said, I would highly support switching to using pyproject.toml over the dumpster fire that is setuptools (or even in conjunction with setuptools if we have to).

As @kwigley says though... we need to have fully separated repos for it to work (I'm looking at you dbt-postgres) well. This has come up elsewhere recently so I made a specific ticket for just that effort.

[github-actions]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please remove the stale label or comment on the issue, or it will be closed in 7 days.

[github-actions]

Although we are closing this issue as stale, it's not gone forever. Issues can be reopened if there is renewed community interest; add a comment to notify the maintainers.

[dbeatty10]

My sense is that we'd need to see our internal software engineers clamoring for Poetry in order to adopt it (which has not happened yet). So whereas I'd see switching to pyproject.toml as highly likely (see #5696), I'd guess switching to Poetry is much less likely for us.