Wondering if poetry / pyproject.toml make sense for Bokeh

[bryevdv]

Hoping some of the experts here can help inform a decision to pursue a switch to Poetry for the Bokeh project. Bokeh is a multi-language project with a complicated build, so it's not entirely clear that the answer is "yes". Thanks in advance for any and all replies!

Background

Our development workflow is heavily tilted toward conda environments. Developing on Bokeh requires several non-python dev-requirements, e.g. NodeJS, LLVM, OpenCV, etc. Getting set up to develop on Bokeh has been reduced to creating one conda env, which has greatly lowered the barrier to entry for new contributors.

So our concerns are not really with using any of the virtual env or locking features of poetry, but really more on improving the packaging and "setup.py develop" (into a conda env) side of things. Specifically:

We would like to have a declarative, single source of truth for specifying runtime dependencies for PyPI and Conda package generation.

We'd like a declarative solution like pyproject.toml because we'd like to automate including some information in our docs and would not like to have to execute a setup.py to scrape it. We'd like to get rid of setup.py completely, if possible.

Some questions:

Conda (build)

Currently we use

{% set data = load_setup_py_data() %}

in our meta.yaml template. This allows us to extract the install_requires and other metadata (e.g. url, license, description) from our setup.py. This is handled by conda-build implicitly without any additional steps.

Question: I found some mentions of poetry and conda-env integrations, ability to turn of virtual env creation, etc. But are there any known integrations for wrt to conda-build e.g automatically supplying a conda recipe with information from pyproject.toml?

BokehJS

The Typescript code for BokehJS has to be compiled to JavaScript, and also bundled in the package. We currently process custom arguments to setup.py

python setup.py develop --build-js   # build BokehJS from scratch
python setup.py develop --install-js # install an already-built BokehJS into the tree

There is a little complication for sdists too. Since building BokehJS requires a toolchain that most folks installing and sdist may not have, we do not include BokehJS source in sdist packages. All sdist ship with a pre-built BokehJS bundled. That said, maybe we can/should just switch to wheels only at this point.

Question: I've seen references to a build.py that can be specified. Is it possible to have more than one "build" script, or to pass arguments to a single script so that different behaviors can be conditioned?

---

Lastly if anyone has any other relevant comments to add, beyond just the questions above, they would be very appreciated.

[NickleDave]

Having a single source of truth would be great.

The question is whether poetry is a good fit for library maintainers, especially in the data science space, who want to let their users work with as many other libraries as possible.

This is distinct from app developers, who want to make sure that their app never breaks in production.
I'm starting to get the feeling that some of the "opinionated" defaults of poetry, and its strict solver, are a result of it being targeted more towards app devs.

Some issues I've run into when switching to poetry that might be relevant for you:

• please stop pinning to major versions by default / forcing SemVer, especially for Python itself #3747
• python_requires="<4" creates issues with dependency solvers nedbat/coveragepy#1020
• Removed python_requires="<4" nedbat/coveragepy#1021
• and related discussion about whether poetry's default of Python "<4.0" is a good idea
  fix: Remove '<4' from python_requires in packaging guide pypa/packaging.python.org#850 (comment)
  https://discuss.python.org/t/use-of-less-than-next-major-version-e-g-4-in-python-requires-setup-py/1066

[NickleDave]

Maybe this issue with libraries v applications is just a general issue for dev tools right now.
pipenv has very specific language in their docs about this:
https://pipenv.pypa.io/en/latest/advanced/#pipfile-vs-setuppy

There is a subtle but very important distinction to be made between applications and libraries. This is a very common source of confusion in the Python community.

Libraries provide reusable functionality to other libraries and applications (let’s use the umbrella term projects here). They are required to work alongside other libraries, all with their own set of sub-dependencies. They define abstract dependencies. To avoid version conflicts in sub-dependencies of different libraries within a project, libraries should never ever pin dependency versions. Although they may specify lower or (less frequently) upper bounds, if they rely on some specific feature/fix/bug. Library dependencies are specified via install_requires in setup.py.

Libraries are ultimately meant to be used in some application. Applications are different in that they usually are not depended on by other projects. They are meant to be deployed into some specific environment and only then should the exact versions of all their dependencies and sub-dependencies be made concrete. To make this process easier is currently the main goal of Pipenv.

To summarize:

For libraries, define abstract dependencies via install_requires in setup.py. The decision of which version exactly to be installed and where to obtain that dependency is not yours to make!
For applications, define dependencies and where to get them in the Pipfile and use this file to update the set of concrete dependencies in Pipfile.lock. This file defines a specific idempotent environment that is known to work for your project. The Pipfile.lock is your source of truth. The Pipfile is a convenience for you to create that lock-file, in that it allows you to still remain somewhat vague about the exact version of a dependency to be used. Pipenv is there to help you define a working conflict-free set of specific dependency-versions, which would otherwise be a very tedious task.
Of course, Pipfile and Pipenv are still useful for library developers, as they can be used to define a development or test environment.
And, of course, there are projects for which the distinction between library and application isn’t that clear. In that case, use install_requires alongside Pipenv and Pipfile.

[bryevdv]

Right, pipenv is defintely not a useful tool for our development case, because locking is not useful for us as library developers. This is also really what I meant about poetry when I said

So our concerns are not really with using any of the virtual env features of poetry

Though perhaps I could have been clearer.

But poetry also has pyproject.toml features, and those are what we are interested in. It's just not clear if there is any easy way to integrate the parts of poetry related to pyproject.toml with our other requirements (conda packaging, basically). It seems perhaps not.

[abn]

I'm starting to get the feeling that some of the "opinionated" defaults of poetry, and its strict solver, are a result of it being targeted more towards app devs.

Wanted to speak to this point briefly. Poetry as a project does not actively prefer one or the other and we intent to support both app and library use cases as equals. When choosing defaults however, unfortunately a choice have to be made and it could be that the app use case was predominant in the user base. But that does not mean that it is set in stone and with the growing community participation, I suspect we will be able to tune these defaults.

[NickleDave]

@bryevdv sorry for dumping my frustrations on you.

Thank you @abn for taking the time to reply.

I should have read what @bryevdv wanted more closely, but my situation is that I do want to use poetry for venv management and for builds -- my code is pure Python.

Some of the issues I raised are just my dumb mistakes 😇 😬

But I think the poetry solver being overly strict is a real issue for anyone trying to use it to develop libraries.
See for example this issue that @nnadeau faced with installing coverage
nedbat/coveragepy#1020
Ideally the solver would be able to resolve the case where a library only wants to lower bound its Python and dependencies, e.g. "Python >3.6", even when one of those dependencies that specifies something like "Python >=3.6, <4.0", by finding the intersection of those two. As @nedbat pointed out.

[abn]

I do not use conda myself, so I cannot comment with any authority on that front.

Question: I found some mentions of poetry and conda-env integrations, ability to turn of virtual env creation, etc. But are there any known integrations for wrt to conda-build e.g automatically supplying a conda recipe with information from pyproject.toml?

I do not think there exists, at present, any integrations to conda-build. However I did find conda/conda#9133, which maybe relevant. Additionally there are projects like https://github.com/dojeda/poetry2conda that you could look at for related examples how conda and poetry might interact.

Based on your example and the definition of load_setup_py_data, I suspect the answer depends on what the execution context is when rendering meta.yaml. If for example, poetry-core is in the context, you can rely on it to extract the metadata, however note that this will only extract what is in the pyproject.toml and not necessarily the locked versions. You could also simply parse the file yourself if toml is present. Alternatively, if the execution context is post poetry build, you maybe able to extract the package metadata using distutilsperhaps.

Maybe you could also contribute something like load_pyproject_poetry_data() to conda-build?

Question: I've seen references to a build.py that can be specified. Is it possible to have more than one "build" script, or to pass arguments to a single script so that different behaviors can be conditioned?

The build scripts can be tricky. Right now you cannot specify arguments to the build script during execution time. The rationale here is that it is expected to be a python script that can be configured to call other scripts as required. We do not have anything planned that might support passing in arguments either since it makes things a bit tricky for such cases when the package build is being executed by a PEP 517 frontend that is not poetry.

There are "task runner" projects building on top of poetry like https://github.com/nat-n/poethepoet that might provide some options, but I would be cautious of adding projects that add poetry itself as a dependency to your dependency tree :)

One final note that might be relevant if you go down the path of building support for cases like this, is the new plugin system. You can see the work in progress on #3733.

Happy to provide any insights I can to help you along with your decision.

[bryevdv]

I do not use conda myself, so I cannot comment with any authority on that front.

If it makes you feel any better, I was the original author of conda in 2012 and I can't comment with authority on its current state either 😀

Thank you very much for the detailed comments. If I may summarize:

• to use pyproject.toml for runtime deps, we would have to hand-template our conda build recipes (at least for now)
• there's not a way to handle our BokehJS build options with poetry right now (but in the future a plugin might be an option)

Then it seems we would not be able to get rid of setup.py at this point, so I think it probably makes sense to simply keep things as they are for the time being. I will raise a conda-build issue regarding load_pyproject_toml_data() and perhaps when the plugin system is completed we can revisit.

[bryevdv]

ref: conda/conda-build#4207