Use pip

[ocefpaf]

Closes conda-forge/staged-recipes#528

PS: b/c conda-forge drifted away from Python standards and does not ship with pip by default we need to specify pip as a build dependency. Ideally we should review that policy and get closer to the Python community and ship pip by default again.

[msarahan]

when did this drift happen? If it is use of conda's add_pip_as_python_dependency setting, then I advised that, and I stand by it. Without it, people very often forget to list setuptools as a dependency. The best solution really is to change the python package, so that it is a metapackage pointing at two things:

1. python itself (no setuptools or pip) - some different package name
2. setuptools and pip, to match upstream python

Recipes should all depend on the package described by 1 here, and never the metapackage. This is the best way to keep all dependencies explicit.

[ocefpaf]

@msarahan I don't know the reason behind this decision and I may be saying something that is absurd here, so bare with me.

I brought this topic b/c I was inquired recently, at a local Python meeting, regarding the "many" things conda-forge is doing that is in the opposite direction of what the larger Python community. I was able to answer most of them but not the lack of pip in out Python package and its use to build packages.

[msarahan]

That sounds like an interesting discussion. I'd be curious to hear those perceptions, and how many of them are conda-forge vs. the broader conda ecosystem (i.e. us). Maybe email me if you don't want to post them here?

[ocefpaf]

That sounds like an interesting discussion. I'd be curious to hear those perceptions, and how many of them are conda-forge vs. the broader conda ecosystem (i.e. us).

Indeed most of them were related to the broader conda ecosystem.

Maybe email me if you don't want to post them here?

I guess I can post here so others can see. It is not a big deal to go off topic on this PR. So... from my memory:

• conda vs wheels: why re-invent the wheel

That one was easy to answer. I explained that conda is older than wheels and how the static linking in wheels are not a good solution for the scientific community.

• conda should be more like pip

Actually I ended up agreeing with this part, not that conda should be more like pip but that pip should be used as much as possible when dealing with pure python packages.

• conda-env is "forking" the community, we should use pipenv

I demonstrated my workflow, that uses conda-envs for reproducible research that depends on R and perk packages. Got a few 👍 from the scientists in the room and a few 👎 from the Python purists.

• conda is diving the Python community

I basically differed them to Jake's post

• conda-forge tries to re-do PyPI

Well... After explaining the differences between conda packages, the agnostic part of coda, etc all over again... I had to agree that when it comes to pure Python packages he was right.

At the end of the day there was nothing in the discussion that we already know, most of the people in the room were already conda users and only a few Python purists that never used it brought those questions.

The only question that I did not had a clean answer was the fact that our Python does not include pip when the Python community is going towards integrating pip. All of our recipes would be 1 line shorter if we did that. I am not sure what are the disadvantages though.

[msarahan]

Well, the history is that conda added (and currently still adds) pip and setuptools to python automatically. That was/is a nasty hack, and it was done at such a low level that it actually introduces some really awkward hacking around the solver level. We'd really like to get rid of it. We acknowledge that upstream python ships with pip, and that's fine. We need a way of having python without pip or setuptools, so that we can package setuptools and pip independently of python. That's why I mentioned the metapackage approach above.

The best thing to tell people, IMHO, is that we do include pip and setuptools with python - but it's as a dependency, not as actual files. If we bundled pip and setuptools with python, we would not be able to update them independently from python. That means old python versions would not get new pip, and we'd also be rebuilding our python packages (very expensive timewise) for the very frequent setuptools releases. Python purists would probably balk at the idea of rebuilding python just to update setuptools, but the packaging model is just fundamentally different. How would they propose updating to the next bugfix revision of python? Maybe virtualenv can do this? Can they update their root installations?

Then you go back to why conda-forge doesn't do the automatic dependency addition: because not listing your actual dependencies makes your software break. Depending on the implicit addition of pip and setuptools is fragile. That implicit addition actually breaks our ability to build setuptools or pip packages, and that's the main reason we turn it off, but having the implicit addition on for building is just generally fragile.

As for forking pypi for pure python stuff, we all feel this pain. It is a goal to have conda working better with wheels, and with pip in general. That's Peter's top priority for conda this year. I don't know what the final solution might be, but perhaps it will include conda directly installing wheels or other contents from pypi. You'll still have the debate between virtualenv and conda envs, but conda would then totally replace pip for conda users. If pip ever gets a dependency solver and supports other languages, then maybe we'll be in a better place to discuss unification of env concepts.

[ocefpaf]

The best thing to tell people, IMHO, is that we do include pip and setuptools with python - but it's as a dependency, not as actual files. If we bundled pip and setuptools with python, we would not be able to update them independently from python. That means old python versions would not get new pip, and we'd also be rebuilding our python packages (very expensive timewise) for the very frequent setuptools releases. Python purists would probably balk at the idea of rebuilding python just to update setuptools, but the packaging model is just fundamentally different. How would they propose updating to the next bugfix revision of python? Maybe virtualenv can do this? Can they update their root installations?

This is what I needed 😄 Thanks!

[ChrisBarker-NOAA]

Just a note here: Python doesn’t bundle pip or setuptools either, for exactly the same reason. It does bundle “ensurepip”, which is a complication. And PyPa is slowly working in disentangling pip and setuptools anyway, so we should keep it clean.

…

-CHB

[ocefpaf]

And PyPa is slowly working in disentangling pip and setuptools anyway,

Interesting, do you have any links to that discussion? I heard the opposite in that meeting.

[CJ-Wright]

@ocefpaf should we post your responses (or something like them somewhere) in the docs? (maybe under FAQ) That way we could have some canned answers for this when people ask.

[ocefpaf]

@ocefpaf should we post your responses (or something like them somewhere) in the docs? (maybe under FAQ) That way we could have some canned answers for this when people ask.

I guess that those would be better in a blog post, after some polishing though, like what Jake did with the conda vs pip myths post.

[ChrisBarker-NOAA]

And PyPa is slowly working in disentangling pip and setuptools anyway, It’s been more or less the primary topic on the distutils SIG for a long while, and there are a few PEPs trying to lock down the meta-Data standards and protocols so we can cleanly separate build tools from install tools. 561 and 566 I think. However, setuptools is a big bundle of building, installing, and run-time tools, so an ugly mess to untangle.

…

-CHB

[pelson]

Let's roll with it. 👍

[pelson]

@jakirkham, given your involvement in #528 and conda-forge/staged-recipes#642, would you mind merging? Given the documentation and the clear statement, I propose we also close conda-forge/staged-recipes#642.

[microe]

I would like to add that even though this change is being suggested for philosophical/community reasons, there is a strong technical argument to suggest the usage of pip over setup tools.

pip, by default will build a wheel instead of an egg. In the case of pure python, a wheel is python version independent (truly universal) and therefore is what 'noarch: python' packages need. Egg builds have an egg-info entry that still contain some python version information. This is only one technical benefit for pure python modules to be wheels instead of eggs.

FWIW, setuptools with wheel can build wheels if so commanded:

python setup.py bdist_wheel --universal;python -m wheel install --force dist/*.whl

This is not nearly as clean as the suggest pip one-liner.

[ChrisBarker-NOAA]

I tried to use pip instead of setuptools for my conda recipes a while back -- in that case, so that I could use the same code to build/install both pure distutils and setuptools based package -- and to get avoid inadvertent use of eggs and eash_install.

I gave up though -- what happened to me is that pip had agressive and confusing (to me) caching that could not seem to turn off -- so when I was testing teh builds, pip would cache something ,and then I'd make a change, and ip would use the cached version, and I wouldn't see the change in the new build. This was a very frustrating experience!

It's a not-issue when builds are done in a clean CI, but could still be a challenge when developing/debugging build recipes...

So we'll see how it goes...

[jakirkham]

Sorry @pelson I think I was out the week that you asked about this and haven't had time to circle back.

If using the old setuptools strategy is problematic, which @microe reasonably contends, then we need to do something different. Using pip seems more straightforward than using the setuptools-based bdist_wheel and should work in more cases (e.g. distutils only packages). Plus it seems to be the way that Python package management is evolving. Given our peculiar relationship with Python (compared to other general purpose package managers) and the reasons already outlined, it seems reasonable to pick a strategy that converges more with the Python community at large. So seems like pip is it.

That said, pip is a package manager itself. So there are some things to keep an eye on. For instance, it would be good if we had some way to block pip from ever downloading other things during a conda-build run regardless of options provided. With setuptools we have a patch that necessarily blocks downloads when running conda-build. The options provided to pip should ensure nothing else gets download. However it still would be nice to have a similar patch for pip if it should be our build tool of choice. That way we can catch issues caused by forgetting the command line options in a particular recipe build or avoid issues caused by generally having missed command line options in our recommendation.

Also there is a rocky history with some packages being installed with pip. While the trend is towards improvement, still not all Python packages play well with pip. So we should remain open-minded in these cases and try other build strategies should they be more reliable or preferred by their project developers.

Other than these two caveats, agree that pip should be encouraged.

[jakirkham]

On a related point, would encourage people interested in using pip to take a look at issue ( pypa/pip#5033 ). Basically pip is planning on implementing build isolation, which would be problematic for conda-build. To correct this, we will need an option to disable build isolation. pip developers have been kind enough to put together PR ( pypa/pip#5053 ) for that intent. Would encourage people to test that patch and provide pip developers and us feedback on their experiences and any issues encountered or suggested improvements. Thanks in advance. :)

[pradyunsg]

it would be good if we had some way to block pip from ever downloading other things

pip has a --no-index option, which disables the use of index servers. I'm not sure how you're blocking downloads when using setuptools but I imagine you'll want to continue with that since pip does need to use setuptools to build packages today. :)

what happened to me is that pip had agressive and confusing (to me) caching that could not seem to turn off

pip also has a --no-cache option to disable the use of caches. :)

PyPa is slowly working in disentangling pip and setuptools anyway,

Yep. PEP 517 and PEP 518.

Also there is a rocky history with some packages being installed with pip.

That's what we're hoping PEP 517 and PEP 518 would help fix. Any constructive feedback on them would definitely be appreciated. :)

---

As for pip's current dependency handling, there's --no-deps to stop pip from installing/looking for dependencies of a package, if that'll make things easier for conda.

Aside: all options in pip have a corresponding environment variable that can be used, if needed. Command line options override the value in the environment variable.

[jakirkham]

Yeah these are all good options to include. Though it becomes a bit much for maintainers to carry around, refresh, and perhaps even no all the ins-and-outs of.

Had looked at making a package in PR ( conda-forge/staged-recipes#642 ) that would automatically set these sorts of things for the user as long as they add it as a build dependency. Other options would be to move this same information into conda-build ( conda/conda-build#2748 ) or pip ( partially pypa/pip#5060 ). Not too concerned which of these options we choose, but it does seem prudent to pick one to keep things maintainable and light for maintainers.

cc @msarahan