Manage dependencies with poetry?

[gjoseph92]

Surely this has come up before, but I couldn't find an issue for it.

Just curious if there's been discussion about using a deterministic, locking package manager like poetry for dask/dask and dask/distributed.

Just thought of it today with the typical spurt of CI failures due to upstream packages releasing new versions: dask/dask#8366 dask/dask#8364 dask/dask#8362

For contributors, it's nice to not have your PR suddenly go red, and be blocked until someone (like @jsignell) makes and merges the changes to work with (or temporarily pin) the new version of a dependency. As a contributor, I also prefer the development experience with a package manager like poetry, since you don't even have to read any "set up your development environment" docs—just run the same command for any poetry project (poetry install --dev) and get the exact same environment as CI.

Of course, we'd still test against new versions of upstream packages. It's just nice when the changes happen deterministically, on your terms (only when you poetry update).

There are other downsides: mindeps isn't automatically supported (python-poetry/poetry#3527), and incompatibilities between dependencies are hard to work around (python-poetry/poetry#697). But just wondering if there's any interest in this.

[jsignell]

To be fair dask/dask#8366 is really just me neglecting upstream failures for several weeks and that action doesn't run on every pull request so it doesn't block anyone.

I am a little worried that if we didn't have so much red CI when an upstream package releases something breaking, it would be easier to miss. But I suppose if we had a nightly job that we closely monitored, then we'd be able to stay on top of released packages that break us.

[jakirkham]

Honestly would rather avoid it. We've had quite a lot of pain in the Conda packaging community with Poetry.

[jrbourbeau]

Yeah, I agree with @jsignell and @jakirkham -- I'd rather focus on increasing visibility/attention of our existing upstream CI build

[jsignell]

I'm going to close this for now butt thank you for bringing it up Gabe. It is always good to step back a bit and make sure we are happy with what we are doing :)