New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add the option to create other sections in Pipfile #726

Closed
marctc opened this Issue Sep 27, 2017 · 22 comments

Comments

Projects
None yet
8 participants
@marctc
Copy link

marctc commented Sep 27, 2017

A common pattern in Django projects is spliting requirements in differents files. We do this in our company following the recommendations of the Two Scoops of Django of @pydanny and @audreyr. In cookiecutter-django you can see this patterns: https://github.com/pydanny/cookiecutter-django/tree/master/%7B%7Bcookiecutter.project_slug%7D%7D/requirements

So, my thought is that pipenv needs to provide more options besides the regular pipenv install and pipenv install --dev. Maybe something like pipenv install requests --section=production and then pipenv install --section=production to install all the production related packages.

@kenneth-reitz

This comment has been minimized.

Copy link
Contributor

kenneth-reitz commented Sep 27, 2017

not happening. explicitly designed for just two groups.

@kenneth-reitz

This comment has been minimized.

Copy link
Contributor

kenneth-reitz commented Sep 27, 2017

e..g this is an explicit design decision, inspired by composer, not an accident.

@kenneth-reitz

This comment has been minimized.

Copy link
Contributor

kenneth-reitz commented Sep 27, 2017

[default] is production.

@marctc

This comment has been minimized.

Copy link
Author

marctc commented Sep 27, 2017

Ok, got it.

I just wanted to separate dev packages from test packages but it's not a big deal.

@kenneth-reitz

This comment has been minimized.

Copy link
Contributor

kenneth-reitz commented Sep 27, 2017

yeah, that's the idea, it's not a big deal. the fact that the composer community has been able to pull this off gives me faith that we'll be able to as well.

keep in mind, the pipfile spec allows for other groups to be added in the future, so we can add this later, if demand is really very high, but so far, it's been smooth sailing.

@kenneth-reitz

This comment has been minimized.

Copy link
Contributor

kenneth-reitz commented Sep 27, 2017

e.g. only having two groups is a superior experience (imo) to having arbitrary groups.

@marctc

This comment has been minimized.

Copy link
Author

marctc commented Sep 27, 2017

keep in mind, the pipfile spec allows for other groups to be added in the future, so we can add this later, if demand is really very high, but so far, it's been smooth sailing.

agree with that

@borfast

This comment has been minimized.

Copy link

borfast commented Oct 13, 2017

Here's a use case for having more than two separate sections: when we want to install certain packages only in production and not in other environments. For example, packages that are only used in production, like gunicorn, or whitenoise, or some email library from a provider like Mailgun, Sendgrid, etc.

@techalchemy

This comment has been minimized.

Copy link
Member

techalchemy commented Oct 13, 2017

Don't you test your production packages?

@pydanny

This comment has been minimized.

Copy link

pydanny commented Oct 13, 2017

If you want to separate requirements this much, think on the following:

  • consider staying with pip and virtualenv. It's low level enough and works great. Especially if you want to keep debugging tools that can be a risk out of production.
  • Remember that standards and practices evolve over time. So maybe pipenv has invalidated the approach we specify in Two Scoops of Django.
@borfast

This comment has been minimized.

Copy link

borfast commented Oct 13, 2017

@techalchemy, sure, I test the production packages, but perhaps I don't want to do it on my local machine and only on a CI server or something.

@pydanny, indeed, things evolve. I did separate the production packages before reading Two Scoops of Django, though :) Great stuff, by the way!

In the end, I agree with what others said: it's not a big deal. It means I will have to install packages I don't need in my local environment but it's not a show-stopper.

@techalchemy

This comment has been minimized.

Copy link
Member

techalchemy commented Oct 14, 2017

@pydanny Pipenv is actually very well suited to keep debugging tools out of production, since if you install them with pipenv install --dev then you have to forcibly and intentionally install your dev dependencies with the --dev flag again in the future. Its just having things that only belong in production that pipenv does not currently offer

@patcon

This comment has been minimized.

Copy link

patcon commented May 8, 2018

Just want to add a use-case of how the two groups are less-than-ideal, although workable:

I would suggest that "groups" are for humans. they are for helping us organize are minds and the projects our minds build. in rubygems land, groups can be arbitrarily named. assuming that prod and dev as the distinction is not something that ecosystem tries to impose.

Example

One project I'm interested in doesn't even use python on the production website, but has some processing tasks that happen locally to build assets. The two groups it wants to make distinction between are "devserver" (a mock api server that runs in python only locally) and "processing" (a bunch of long-compiling scientific packages for doing geospatial work). Forcing a project like this in the dev/prod dichotomy makes less sense.

Conclusion

Basically, I feel that the distinction is more opinionated than need be, and prevents the tool from speaking human in the way that users might like it to. In my example, we need to have docs (in the CLI instructions and the code comments) to explain this, instead of just having intuitive names of the groups that can go in a command line flag :)

Anyhow, thanks so so much for all your hard work. Just wanted to provide a counter-point. Everything is work-aroundable, but I simply would contest that it shouldn't need to be -- it seems a reasonable ask. (fwiw, as someone who comes from php land, i feel rubygems and maybe go is the package management ecosystem to emulate, not composer)

Y'all are awesome! ❤️

@kenneth-reitz

This comment has been minimized.

Copy link
Contributor

kenneth-reitz commented May 8, 2018

Composer has been successful with just two groups. That's what inspired this decision.

@borfast

This comment has been minimized.

Copy link

borfast commented May 9, 2018

I also worked with PHP for a very long time and, just like Patrick, don't really think it's the best thing to emulate.

@kennethreitz, how would Composer handle the scenario @patcon described?

@uranusjr

This comment has been minimized.

Copy link
Member

uranusjr commented May 9, 2018

I’m not very involved in the PHP community, but this seems to sum up their opinions: https://matthiasnoback.nl/2014/04/theres-no-such-thing-as-an-optional-dependency/

And really I agree with it. When developing an application (which is what Pipenv is for), the dev/prod combination is theoretically enough, and usually you even only need one.

In practice, however, I personally think additional groups are handy to have, especially if you need to use services that may not be able to install everything. For example I might be building a website using a Python static site generator, with some documentation written with Sphinx, use some plugins to auto-generate some stuff locally, a test runner to make sure the plugins work correctly, and a task runner to help automating the deployment. But not every service can install everything, especially specialised ones.

It is not impossible to work around (requirements files are always available), but this means my environments are not as reproducible as a straight Pipenv-managed one can be. I feel Python is used for too many things (compared to, say, PHP) that conditional deployment could be a handy thing to have. Not theoretically necessary, but also not totally redundant.

@kenneth-reitz

This comment has been minimized.

Copy link
Contributor

kenneth-reitz commented May 9, 2018

My ethos is "solve the 95% use-case". This is the 95% use-case.

@gsemet

This comment has been minimized.

Copy link
Contributor

gsemet commented May 9, 2018

pipenv is used on real life with just two groups and it works great. default group only describe the running dependencies, while all the doc/api autogenation, lint, checks, unit tests dependencies, belongs to the dev group.
The only reason to add more group is that if there were incompatibility in two dev dependencies. For example the sphinx generator and pylint has major, unsolvable dependencies issue. I never saw that in real life.

@borfast

This comment has been minimized.

Copy link

borfast commented May 9, 2018

What I don't get is why impose a limitation that brings no clear advantage (aside from the obvious simpler implementation but I wouldn't say it's such a big difference), when lifting that limitation would make life easier for many people, even if they are just 5%. But hey, it's not my project and I don't want to become a PITA by insisting on this ad nauseam :)

@techalchemy

This comment has been minimized.

Copy link
Member

techalchemy commented May 9, 2018

The implementation factor is a huge advantage though. As a maintainer I can say that anywhere our codebase is complex or hard to follow I’m reluctant to accept any PR I didn’t personally test by hand and trace just because things are much more likely to break. Too much code like that and the project becomes a maintenance nightmare.

For end users, if you’re new to python and you see that you can have 5 sections you start thinking maybe you should. Then every time you try to use pipenv you can’t remember what section you put a package in. You don’t use cat Pipfile because you’re new, you have to open it in an editor and this is a slow process. You get frustrated. Eventually you just start guessing and accidentally wind up installing the same package in a brand new section you just created. Now you have 6 sections. Ultimately you decide the whole process is overly complicated and you don’t feel like dealing with it anymore so you go back to some other thing that isn’t pipenv or maybe it isn’t even python.

@uranusjr

This comment has been minimized.

Copy link
Member

uranusjr commented May 9, 2018

Yes, implementation is a big factor. As a contributor I went through multiple episodes that went like “maybe I should come up with something to try persuading Kenneth and Dan” and after 15 minutes my head started hurting so I stopped. For anyone strongly pushing this issue, I encourage you to get yourself familiar this whole dependency resolution thing is done, and try to come up with a workable implementation. Hopefully one of us will eventually come up with something, but before that, take my words and drop it, this conversation won’t go anywhere.

@borfast

This comment has been minimized.

Copy link

borfast commented May 9, 2018

I totally agree with the implementation complexity weighing considerably on the decision of implementing something. Not so sure about the multiple sections, though. But yes, it's dropped :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment