Pipenv Support

[sbeleidy]

Are there any plans to support pipenv?

[FlorianWilhelm]

Hi, so far I haven't used it. Supporting it would mean to provide a basic Pipfile, right? Should be no problem with the new extension system. A --pipenv flag would then create a basic Pipfile instead of requirements.txt.
Do you want to provide a PR?

[sbeleidy]

I think it's a little more than that. Adding a Pipfile is a simple thing - the issue I'm not sure how to deal with is loading the dependencies from the Pipfile to be used in setup.py. When you install the package it should install all its dependencies as well.

Happy to provide a PR if you can direct me to the best way to deal with this.

[FlorianWilhelm]

@sbeleidy, and don't think that setup.py should read the requirements from Pipefile. The requirements in the Python package defined by setup.cfg are different than the requirements needed to set up and run the project itself. Therefore I think an extension should just write a Pipfile with some sane defaults instead of the current requirements.txt.

[sbeleidy]

@FlorianWilhelm - I think I'm a little confused. If your package has dependencies, you are setting those in setup.cfg? Do you have an example of that?

I thought it would read the requirements.txt and add those packages.

[FlorianWilhelm]

Sure, this behaviour was changed in version 3.0 of PyScaffold. As you can see in this configuration example there is a key install_requires where you would list the unpinned, direct requirements of your package.

The requirements.txt itself it meant to help you set up your development or runtime environment pretty fast and there you would rather use pinned versions of your requirements and maybe even add the requirements of your requirements and so on to make sure the environment is reproducible. You would thereby use tools like pip-tools to manage your requirements.txt. The new Pipfile format replaces the old-school requirements.txt and is managed by pipenv.

Maybe @HolgerPeters, @sebastianneubauer or @dhepper are more experienced with pipenv and can comment on that.

[HolgerPeters]

I think it's a little more than that. Adding a Pipfile is a simple thing - the issue I'm not sure how to deal with is loading the dependencies from the Pipfile to be used in setup.py. When you install the package it should install all its dependencies as well.

I think the best bet at the moment is to do double book keeping for Pipfile and install_requires in setup.py. What I mean is that they are actually two similar, but subtly different lists. Pipfile's purpose is to generate a pinned list of requirements needed to operate / run and develop a package. install_requires defines a minimum set of typical requirements, not necessarily listing all of them, for example if some code paths aren't intended to be used. Especially if you are writing a library, be aware that by listing dependencies in install_requires you are forcing them onto your consumer, not always the best option.

To sum it up, I would go about this as follows:

• If you write an application, don't use install_requires at all and set up your deployment using pipenv.
• If you write a library, do the extra effort and maintain install_requires and Pipfile.

I'd like to add that pipenv is a major improvement over the pip-tools workflow I used to favour. Especially since it has a sane way of handling development packages --dev. (Sometimes it is a bit too magic for my taste, for example pypa/pipenv#1319 , but overall a great experience and worth potential additional effort in double bookkeeping.

[FlorianWilhelm]

@sbeleidy Has this helped your understanding? One could still add a default Pipfile to PyScaffold and it would be pretty easy I guess.

[abravalheri]

@HolgerPeters do you mind to clarify me some extra bits?

For example: isn't the difference you have described exactly the point of having the --dev flag and the dev-packages section?

Theoretically, the library developer could put the minimal requirements in the packages section with the most generic versions possible, and in the dev-requirements the ones that are used for code paths not intended for public consumption (as you have explained)... If the dev includes the lock in the repository, the virtualenv for development is then perfectly reproductible, but at the same time PyScaffold could read the raw Pipenv file (just the main section, without the dev) to determine the minimal generic requirements for the lib...

[abravalheri]

The problem that I see with double bookkeeping is that it scales very fast...

Right now for example, I am having some trouble because I have to keep the test requirements in sync in the setup.cfg and in tox.ini files. So if you add a Pipfile on top of it you end up with at least 3 files to keep in sync...

The advantage of pip-tools, is that it is backwards compatible and therefore supported out-of-the box by tools like tox.

So in the context of the example I was describing, if PyScaffold could support reading the test requirements from the test-requirements.in instead of directly from setup.cfg, I could use -r test-requirements.txt in tox togheter with pip-tools to achieve single source of truth...

The same thing for the regular dependencies: PyScaffold could read requirements.in to generate the dependencies for setuptools, while the dev could add requirements.txt to the repository to guarantee reproducibility.

And if someone would like to have special dependencies for some corner use cases, that person could add a third dev-dependencies.in/dev-dependencies.txt pair...

[abravalheri]

Well, I did some research about how to achieve "single source of truth" while using tools like Pipenv and tox. The best practices I could find so far are:

1. Rely on install_requires (setup.{py,cfg}) and make Pipenv read it by adding something like "-e ." - ref1, ref2

2. Sadly ignore test_requires since tox don't read it. The solution instead is to create optional dependency sections (e.g. "dev" to be compliant with the Pipenv way, see ref2) and use them in the tox.ini file through the extras key - ref3.

(But the second practice can break python setup.py test though...)

[abravalheri]

Hi @FlorianWilhelm, please find bellow my current proposal:

Summary

This proposal is a pragmatic compromise that enables developers to use tools (e.g. pipenv or pip-tools) for managing concrete requirements, while keeping sanity when using setuptools for packing and tox or pytest-runner for testing.
The primary goal is to have clear separation and good management of abstract and concrete requirements, while avoiding double book keeping.
In a nutshell, it consist of finding a middle ground, respected by all the aforementioned tools:

• the install_requires option in setup.cfg is used as the single source of truth for abstract requirements (more generic and permissive)
• the tests_requires option in setup.cfg is ignored. Instead a section (e.g. testing) in extras_require is used as the single source of truth for the test dependencies. These dependencies are not directly installed/used by the user and just exist in the test virtualenvs
• an automatically generated file (e.g. Pipfile.lock or (dev-)requirements.txt) is used for keeping the concrete requirements and therefore allows repeatable installations
• intermediate files (e.g. Pipfile or (dev-)requirements.in) are used to proxy setup.cfg install_requirements to external tools. Additionally, these files can include other packages used in the dev environment (e.g. interactive interpreters, profilers, test runners, etc...)

Motivation

PyScaffold aims the development of Python projects that can be easily packaged and distributed using the standard PyPI and pip flow, which includes support for dependency installation. In the pip ecosystem two approaches for specifying those dependencies can be followed:

• the first approach consists of having very permissive, minimal and generic dependencies, with versions specified by ranges (abstract), so anyone can install the package without many conflicts, sharing and reusing as much as possible dependencies that are already installed/also required by other packages
• the second approach consists of having very strict dependencies, with pinned versions (concrete), so all the users will have repeatable installations

Both approaches have advantages and disadvantages, and usually are used together in different phases of a project. For more information about this topic check this post.

Several tools have being used to manage the concrete vs abstract requirements scenario, among them very popular ones are: pipenv and pip-tools. Since these tools have their own mechanism of declaring dependencies, it is not straightforward to image how to use them without copying and pasting from setup.cfg and incur in the trouble of manually keeping all the files in sync.

Additionally, tox -one of the most famous test runners for Python- adds yet another file where dependencies can be declared, and completely ignores setuptools tests_require standard, which makes the manual synchronization even harder.

The proposal is to accept the current limitations of all the mentioned tools, but create a workflow that, although imperfect, can handle the vast majority of the use cases (with some user understanding).

Detailed Solution

The current limitations and workarounds considered are:

• setuptools: we cannot read requirements from separated files into install_requires via setup.cfg syntax - ref
• tox uses setuptools' install-require but doesn't respect tests_require. However, its documentation suggests the usage of a testing group in extras_required - ref, ref.
• pipenv doesn't support extra requirements on editable dependencies - ref, ref
• pytest-runner doesn't support specifying a single group of extra requirements. Instead all the extra requirements are installed at once - ref
• pipenv just support 2 dependency groups: "production" and dev - ref, ref

Therefore, we can split the problem in 3 main components:

1. How to declare package dependencies in a way it can be re-used by setuptools, pipenv and pip-tools without double book keeping?
2. How to declare test dependencies in a way it can be re-used by tox and pytest-runner without double book keeping?
3. How to declare dev-only dependencies?

Note that:

• for problem 1, we don't have to be concerned with the test runners, since use setuptools under the hood;
• for problem 2, we don't have to be concerned with setuptools and the concrete dependencies managers, since the test runners are completely able to run the tests in isolated environments, installing test dependencies on the fly (as long as the user have the test runner installed everything is fine - setuptools don't really enforces test_requirements...)
• for problem 3, since the dev dependencies are not essential, they don't have to be present in setup.cfg or the test configuration...

For problem 1, the best solution is to use setup.cfg as single source of truth and rely on pipenv install -e . or echo '-e .' >> requirements.in. This way we can keep abstract requirements when defining the package (so minimal and flexible installations are performed), while pipenv and pip-tools still can be used to compile concrete requirements. In turn Pipfile.lock or (dev-)requirements.txt can be added to the version control system allowing the devs to create repeatable environments.

For problem 2, we completely ignore tests_requires since tox doesn't support it and rely on a testing group of extra requirements, by adding extras = testing to tox configuration and extras = True to pytest-runner configuration. As a drawback, we cannot restrict extras installation in pytest-runner to just testing, however one could argue that:

• extra requirements are not that common...
• the system should be tested with all the extra requirements anyway...
• if the project requires different combinations of requirements for testing, a more sophisticated tool like tox would end up being used anyway...

Although these arguments are a little bit weak, they make some sense, and, as far as pragmatism goes, this drawback is an acceptable price to be paid for reasonable defaults.

For problem 3, we can just use the concrete requirements management tools, since the testing and packaging environment should be agnostic to dev-only tools.

Therefore, the proposed workflow is:

• Add abstract dependencies to setup.cfg
• Proxy setup.cfg by doing pipenv install -e . [Pipenv] or echo "-e ." >> requirements.in [pip-tools]
• Add dev dependencies by doing pipenv install -d XXXX [Pipenv] or echo "XXXX" >> dev-requirements.in [pip-tools]
• Use pipenv update -d [Pipenv] or pip-compile --output-file (dev-)requirements.txt (dev-)requirements.in + (source /path/to/venv/bin/activate &&) pip-sync requirements.txt dev-requirements.txt [pip-tools] to resolve/install concrete dependencies
• Add Pipfile.lock [Pipenv] or (dev-)requirements.txt to source control for repeatable installations
• Use pipenv run YYYY [Pipenv] or (source /path/to/venv/bin/activate &&) YYYY [pip-tools] to run commands inside the venv (e.g. pipenv run tox)
• Don't expose test requirements directly to Pipenv/pip-tools. Instead, just rely on tox/pytest-runner to install them inside the test venv.

The repository abravalheri/dummy-pipenv-example-for-pyscaffold is a proof of concept, and presents a project where this methodology is applied (see branches pipenv and pip-tools).

Migration plan

1. Add a deprecation warning when after generating requirements.txt during a putup
2. Add documentation about how to use pip-tools and Pipfile together with PyScaffold
3. Move test dependencies to testing under extras_require in the setup.cfg template
4. Add option extras = True in the section [test] of the setup.cfg template
5. Drop requirements.txt files from PyScaffold core after 4.0
6. Optionally create extensions pyscaffoldext-pipenv, pyscaffoldext-piptools and link them in the project page

How do we teach this?

As mentioned in the previous section, we should add documentation about how to use Pipfile and pip-tools with PyScaffold, in a way the single source of truth principle is followed. Some text similar to this proposal can be use (but we don't have to be so detailed about limitations).

Optionally, we can have a Best Practices section in the main website, where we describe a "blessed" approach of using PyScaffold, recommended extensions, etc...

Drawbacks

With this approach we completely ignore tests_requires standard of setuptools and install make pytest-runner install dependencies inside the test venv that can be unrelated to testing (these drawbacks were already discussed in the details section).

Moreover, since both setuptools and pipenv are changing, targeting the new pyproject.toml standard and are both driven by PyPA, they might change in the future. In this case, we should have to review PyScaffold approach/implementation to be compatible.

Alternatives

There are other tools for packaging and venv management mentioned in the Python Packaging Guide and in some articles, and in this case a different research would be required. But I would prefer sticking with PyPA...

We can also just choose one (pip-tools, Pipenv).

[FlorianWilhelm]

@abravalheri First of all, thanks for this great proposal, I have never seen something like this in a Github issue discussion before. It's on the same level of quality and detail as a PEP.

Overall I think this is a really good and pragmatic approach that we should follow for a version 4.0 respecting Semantic Versioning. Regarding the drawbacks of not using the test_requires standard, I think that it will be deprecated soon anyway as it hasn't seen any major adoption over a long period of time. Managing test requirements should be the responsibility of the test runner like tox. One could even argue that one day we should drop pytest-runner since running py.test directly does about the same und there is no real advantage of using python setup.py test instead.

Your migration sounds also like a great fit for those changes. Regarding pip-tools vs. Pipenv, I wonder if we should maybe support only one (also to avoid much extra work). I haven't not yet really tried them, mainly because I use a lot of conda, and thus I wonder if we are already now able to tell which one of them will make it in the end. From what I read Pipenv has more traction right now and it seems more sophisticated. What are do you think? Should we start with one and wait until people ask for the other? Which would that be?

I am also not sure right now if we should have a pyscaffoldext-pipenv/piptools extension or if we should rather integrate it into core. My gut feeling says that having it as an extension makes more sense as long as there is no established standard and also many people use conda and might be irritated if we promote pipenv/piptools too much and add their files by default. What would be your preferred way?

One thing I am 100% sure about is that we should no longer generate a requirements.txt from version 4.0 on and document the usage of pipenv/pip-tools better. One other thing is the deprecation warning for requirements.txt starting on from version 3.1. Deprecation warnings are great for libraries but wouldn't we irritate people when we show them a deprecation warning after they have just invoked putup the right way?

Anyways, let's move further into this direction. I guess it's best to keep it in a separate v4.x branch here or your local repo for now until we released version 3.1.

[abravalheri]

Hi @FlorianWilhelm, thank you very much for your comments. One thing that might be not clear yet is that I believe we can do the steps 1-4 of the migration plan already for the new minor version of PyScaffold, without having to wait for v4.

Basically, in my opinion, we don't break any documented behaviour (in other words: pytest-runner will still respect test_requires if the user decides to add dependencies there, and if there is anyone manually managing packages with requirements.txt, there will be nothing preventing it to work). We only adopt and document new best practices, keeping everything backwards compatible.

Instead of warnings, we could add comments on top of requirements.txt about the file not being generated in the next major release and pointing the user to the documentation of dependency management (to be written), where we say that abstract dependencies should be kept in setup.cfg while concrete dependencies can be done manually or using tools like pipenv. Something similar can be done for test_requires.

Regarding creating extensions for the tools themselves, I believe that, for now, if PyScaffold is compatible out-of-the-box with both, and provides good documentation about how to adopt them, it might be already a good improvement.

We can monitor the interest of the community and hold the decision, or even adopt under the PyScaffold organization extensions that might eventually appear. But of course, if we see in this topic an opportunity to incentivise good practices and decide to do something soon, I would go for an extension (the less files under the same git repo, the easier is to have contributions/maintain/evolve) and say we try Pipenv first (basically because it is promoted by PyPA, so the chances of becoming a de facto standard are very high).

Finally, regarding the roadmap, maybe we can try to avoid the fragmentation between the versions. For v4.x, considering that steps 1-4 can be implemented already, we can just rely on TODO notes to remember to exclude requirements.txt when the time comes... What do you think?

[FlorianWilhelm]

@abravalheri, you got some very good points. I also like your solution with adding a comment into requirements.txt instead of a warning on the command line. Also your points regarding the fragmentation are convincing, so let's directly integrate steps 1-4 into the current master.

Also thumps up for a Pipenv extension that is recommended in PyScaffold's documentation.

I see only one thing we got to consider. While we are preparing the v3.1 release and update the documentation we got to make sure that the displayed default documentation is still one that is compatible with the current v3.0.x release. Maybe it's best to create a v3.0.x git branch and let readthedocs pull from there by default.

[abravalheri]

@FlorianWilhelm, it seems that RTD already do this job for us... as long as we don't create any tagged release, the documentation for the stable version will be the one pointing to the latest tag. Seems to be pretty straightforward to me.

[abravalheri]

Is this issue solved by #188? (In theory with that PR, PyScaffold does support if the user wants to use Pipenv, despite not generating the Pipfile itself)

[FlorianWilhelm]

With the added docs about dependency management I think this could be closed. Adding a dummy Pipfile really has not much benefits right now and could later be added with the help of an extension.

[abravalheri]

Sounds fair to me! Before v3.1 I would like just to mark this feature as experimental in the docs (since Pipenv is moving fast and probably breaking things while still trying to figure out an stable API, it is best to not commit so soon).

[abravalheri]

Sorry for the delay. I think with #209 we can finally close this issue, without fear of breaking changes in the future.

[FlorianWilhelm]

@abravalheri, thanks! Great, another issue moving to done :-)