Pipenv Support

[sbeleidy]

Are there any plans to support pipenv?

[FlorianWilhelm]

Hi, so far I haven't used it. Supporting it would mean to provide a basic Pipfile, right? Should be no problem with the new extension system. A --pipenv flag would then create a basic Pipfile instead of requirements.txt.
Do you want to provide a PR?

[sbeleidy]

I think it's a little more than that. Adding a Pipfile is a simple thing - the issue I'm not sure how to deal with is loading the dependencies from the Pipfile to be used in setup.py. When you install the package it should install all its dependencies as well.

Happy to provide a PR if you can direct me to the best way to deal with this.

[FlorianWilhelm]

@sbeleidy, and don't think that setup.py should read the requirements from Pipefile. The requirements in the Python package defined by setup.cfg are different than the requirements needed to set up and run the project itself. Therefore I think an extension should just write a Pipfile with some sane defaults instead of the current requirements.txt.

[sbeleidy]

@FlorianWilhelm - I think I'm a little confused. If your package has dependencies, you are setting those in setup.cfg? Do you have an example of that?

I thought it would read the requirements.txt and add those packages.

[FlorianWilhelm]

Sure, this behaviour was changed in version 3.0 of PyScaffold. As you can see in this configuration example there is a key install_requires where you would list the unpinned, direct requirements of your package.

The requirements.txt itself it meant to help you set up your development or runtime environment pretty fast and there you would rather use pinned versions of your requirements and maybe even add the requirements of your requirements and so on to make sure the environment is reproducible. You would thereby use tools like pip-tools to manage your requirements.txt. The new Pipfile format replaces the old-school requirements.txt and is managed by pipenv.

Maybe @HolgerPeters, @sebastianneubauer or @dhepper are more experienced with pipenv and can comment on that.

[HolgerPeters]

I think it's a little more than that. Adding a Pipfile is a simple thing - the issue I'm not sure how to deal with is loading the dependencies from the Pipfile to be used in setup.py. When you install the package it should install all its dependencies as well.

I think the best bet at the moment is to do double book keeping for Pipfile and install_requires in setup.py. What I mean is that they are actually two similar, but subtly different lists. Pipfile's purpose is to generate a pinned list of requirements needed to operate / run and develop a package. install_requires defines a minimum set of typical requirements, not necessarily listing all of them, for example if some code paths aren't intended to be used. Especially if you are writing a library, be aware that by listing dependencies in install_requires you are forcing them onto your consumer, not always the best option.

To sum it up, I would go about this as follows:

• If you write an application, don't use install_requires at all and set up your deployment using pipenv.
• If you write a library, do the extra effort and maintain install_requires and Pipfile.

I'd like to add that pipenv is a major improvement over the pip-tools workflow I used to favour. Especially since it has a sane way of handling development packages --dev. (Sometimes it is a bit too magic for my taste, for example pypa/pipenv#1319 , but overall a great experience and worth potential additional effort in double bookkeeping.

[FlorianWilhelm]

@sbeleidy Has this helped your understanding? One could still add a default Pipfile to PyScaffold and it would be pretty easy I guess.

[abravalheri]

@HolgerPeters do you mind to clarify me some extra bits?

For example: isn't the difference you have described exactly the point of having the --dev flag and the dev-packages section?

Theoretically, the library developer could put the minimal requirements in the packages section with the most generic versions possible, and in the dev-requirements the ones that are used for code paths not intended for public consumption (as you have explained)... If the dev includes the lock in the repository, the virtualenv for development is then perfectly reproductible, but at the same time PyScaffold could read the raw Pipenv file (just the main section, without the dev) to determine the minimal generic requirements for the lib...

[abravalheri]

The problem that I see with double bookkeeping is that it scales very fast...

Right now for example, I am having some trouble because I have to keep the test requirements in sync in the setup.cfg and in tox.ini files. So if you add a Pipfile on top of it you end up with at least 3 files to keep in sync...

The advantage of pip-tools, is that it is backwards compatible and therefore supported out-of-the box by tools like tox.

So in the context of the example I was describing, if PyScaffold could support reading the test requirements from the test-requirements.in instead of directly from setup.cfg, I could use -r test-requirements.txt in tox togheter with pip-tools to achieve single source of truth...

The same thing for the regular dependencies: PyScaffold could read requirements.in to generate the dependencies for setuptools, while the dev could add requirements.txt to the repository to guarantee reproducibility.

And if someone would like to have special dependencies for some corner use cases, that person could add a third dev-dependencies.in/dev-dependencies.txt pair...

[abravalheri]

Well, I did some research about how to achieve "single source of truth" while using tools like Pipenv and tox. The best practices I could find so far are:

1. Rely on install_requires (setup.{py,cfg}) and make Pipenv read it by adding something like "-e ." - ref1, ref2

2. Sadly ignore test_requires since tox don't read it. The solution instead is to create optional dependency sections (e.g. "dev" to be compliant with the Pipenv way, see ref2) and use them in the tox.ini file through the extras key - ref3.

(But the second practice can break python setup.py test though...)

[abravalheri]

Hi @FlorianWilhelm, please find bellow my current proposal:

Summary

This proposal is a pragmatic compromise that enables developers to use tools (e.g. pipenv or pip-tools) for managing concrete requirements, while keeping sanity when using setuptools for packing and tox or pytest-runner for testing.
The primary goal is to have clear separation and good management of abstract and concrete requirements, while avoiding double book keeping.
In a nutshell, it consist of finding a middle ground, respected by all the aforementioned tools:

• the install_requires option in setup.cfg is used as the single source of truth for abstract requirements (more generic and permissive)
• the tests_requires option in setup.cfg is ignored. Instead a section (e.g. testing) in extras_require is used as the single source of truth for the test dependencies. These dependencies are not directly installed/used by the user and just exist in the test virtualenvs
• an automatically generated file (e.g. Pipfile.lock or (dev-)requirements.txt) is used for keeping the concrete requirements and therefore allows repeatable installations
• intermediate files (e.g. Pipfile or (dev-)requirements.in) are used to proxy setup.cfg install_requirements to external tools. Additionally, these files can include other packages used in the dev environment (e.g. interactive interpreters, profilers, test runners, etc...)

Motivation

PyScaffold aims the development of Python projects that can be easily packaged and distributed using the standard PyPI and pip flow, which includes support for dependency installation. In the pip ecosystem two approaches for specifying those dependencies can be followed:

• the first approach consists of having very permissive, minimal and generic dependencies, with versions specified by ranges (abstract), so anyone can install the package without many conflicts, sharing and reusing as much as possible dependencies that are already installed/also required by other packages
• the second approach consists of having very strict dependencies, with pinned versions (concrete), so all the users will have repeatable installations

Both approaches have advantages and disadvantages, and usually are used together in different phases of a project. For more information about this topic check this post.

Several tools have being used to manage the concrete vs abstract requirements scenario, among them very popular ones are: pipenv and pip-tools. Since these tools have their own mechanism of declaring dependencies, it is not straightforward to image how to use them without copying and pasting from setup.cfg and incur in the trouble of manually keeping all the files in sync.

Additionally, tox -one of the most famous test runners for Python- adds yet another file where dependencies can be declared, and completely ignores setuptools tests_require standard, which makes the manual synchronization even harder.

The proposal is to accept the current limitations of all the mentioned tools, but create a workflow that, although imperfect, can handle the vast majority of the use cases (with some user understanding).

Detailed Solution

The current limitations and workarounds considered are:

• setuptools: we cannot read requirements from separated files into install_requires via setup.cfg syntax - ref
• tox uses setuptools' install-require but doesn't respect tests_require. However, its documentation suggests the usage of a testing group in extras_required - ref, ref.
• pipenv doesn't support extra requirements on editable dependencies - ref, ref
• pytest-runner doesn't support specifying a single group of extra requirements. Instead all the extra requirements are installed at once - ref
• pipenv just support 2 dependency groups: "production" and dev - ref, ref

Therefore, we can split the problem in 3 main components:

1. How to declare package dependencies in a way it can be re-used by setuptools, pipenv and pip-tools without double book keeping?
2. How to declare test dependencies in a way it can be re-used by tox and pytest-runner without double book keeping?
3. How to declare dev-only dependencies?

Note that:

• for problem 1, we don't have to be concerned with the test runners, since use setuptools under the hood;
• for problem 2, we don't have to be concerned with setuptools and the concrete dependencies managers, since the test runners are completely able to run the tests in isolated environments, installing test dependencies on the fly (as long as the user have the test runner installed everything is fine - setuptools don't really enforces test_requirements...)
• for problem 3, since the dev dependencies are not essential, they don't have to be present in setup.cfg or the test configuration...

For problem 1, the best solution is to use setup.cfg as single source of truth and rely on pipenv install -e . or echo '-e .' >> requirements.in. This way we can keep abstract requirements when defining the package (so minimal and flexible installations are performed), while pipenv and pip-tools still can be used to compile concrete requirements. In turn Pipfile.lock or (dev-)requirements.txt can be added to the version control system allowing the devs to create repeatable environments.

For problem 2, we completely ignore tests_requires since tox doesn't support it and rely on a testing group of extra requirements, by adding extras = testing to tox configuration and extras = True to pytest-runner configuration. As a drawback, we cannot restrict extras installation in pytest-runner to just testing, however one could argue that:

• extra requirements are not that common...
• the system should be tested with all the extra requirements anyway...
• if the project requires different combinations of requirements for testing, a more sophisticated tool like tox would end up being used anyway...

Although these arguments are a little bit weak, they make some sense, and, as far as pragmatism goes, this drawback is an acceptable price to be paid for reasonable defaults.

For problem 3, we can just use the concrete requirements management tools, since the testing and packaging environment should be agnostic to dev-only tools.

Therefore, the proposed workflow is:

• Add abstract dependencies to setup.cfg
• Proxy setup.cfg by doing pipenv install -e . [Pipenv] or echo "-e ." >> requirements.in [pip-tools]
• Add dev dependencies by doing pipenv install -d XXXX [Pipenv] or echo "XXXX" >> dev-requirements.in [pip-tools]
• Use pipenv update -d [Pipenv] or pip-compile --output-file (dev-)requirements.txt (dev-)requirements.in + (source /path/to/venv/bin/activate &&) pip-sync requirements.txt dev-requirements.txt [pip-tools] to resolve/install concrete dependencies
• Add Pipfile.lock [Pipenv] or (dev-)requirements.txt to source control for repeatable installations
• Use pipenv run YYYY [Pipenv] or (source /path/to/venv/bin/activate &&) YYYY [pip-tools] to run commands inside the venv (e.g. pipenv run tox)
• Don't expose test requirements directly to Pipenv/pip-tools. Instead, just rely on tox/pytest-runner to install them inside the test venv.

The repository abravalheri/dummy-pipenv-example-for-pyscaffold is a proof of concept, and presents a project where this methodology is applied (see branches pipenv and pip-tools).

Migration plan

1. Add a deprecation warning when after generating requirements.txt during a putup
2. Add documentation about how to use pip-tools and Pipfile together with PyScaffold
3. Move test dependencies to testing under extras_require in the setup.cfg template
4. Add option extras = True in the section [test] of the setup.cfg template
5. Drop requirements.txt files from PyScaffold core after 4.0
6. Optionally create extensions pyscaffoldext-pipenv, pyscaffoldext-piptools and link them in the project page

How do we teach this?

As mentioned in the previous section, we should add documentation about how to use Pipfile and pip-tools with PyScaffold, in a way the single source of truth principle is followed. Some text similar to this proposal can be use (but we don't have to be so detailed about limitations).

Optionally, we can have a Best Practices section in the main website, where we describe a "blessed" approach of using PyScaffold, recommended extensions, etc...

Drawbacks

With this approach we completely ignore tests_requires standard of setuptools and install make pytest-runner install dependencies inside the test venv that can be unrelated to testing (these drawbacks were already discussed in the details section).

Moreover, since both setuptools and pipenv are changing, targeting the new pyproject.toml standard and are both driven by PyPA, they might change in the future. In this case, we should have to review PyScaffold approach/implementation to be compatible.

Alternatives

There are other tools for packaging and venv management mentioned in the Python Packaging Guide and in some articles, and in this case a different research would be required. But I would prefer sticking with PyPA...

We can also just choose one (pip-tools, Pipenv).