use conda to get Python, even for requirements.txt

[minrk]

Using conda to get Python itself would simplify supporting multiple Python versions (#146) for the pure Python requirements.txt case. It would also simplify things because then requirements.txt and environment.yml would only differ by one line: whether to use conda env update -f or pip install -r to install the user's files.

This will allow us to easily support many Python versions without relying on what's packaged in the base distro.

[minrk]

It would also eliminate the maintenance of multiple base environments, since the base packages would still be the same environment.yml files.

[yuvipanda]

I'm going to try to consolidate my previously fragmented and maybe not very productive comments into one here. Apologies for the spam!

There are two underlying problems from what I can see:

1. Decoupling version of Python used from version provided by underlying distro. This allows us to support a wider range of Python versions across time than any distro would support. This is important both for reproducibility (older version of python support) and coolness (newer versions of python support). We could get these from conda instead of apt, but I prefer the approach in [WIP] Use pyenv to manage base python versions #190 to solve this.
2. Maintenance burden to us maintainers of maintaining two base environments - for conda and pip. IMO, this is preferable to mixing conda and pip. Something like document and/or automate updating frozen environments #188 or https://greenkeeper.io/ (but for py dependencies) should help with this.

Does that cover it?

[minrk]

The other bit is that I don't think we should be recompiling Python itself, and pyenv does that, both for maintenance reasons (building Python yourself, it is easy to get into misconfigured situations like missing sqlite or ssl issues) and time (the venv part of the tests in the pyenv PR were aborted by travis because they now take 49 minutes instead of 4). So I think for venv we should be caching and staging binary builds rather than letting pyenv do its builds, and doing that seems like it might be more complicated than initializing Python (if nothing else) with conda.

There aren't really issues using conda and pip together, as long as pip comes after conda. Especially if you start very basic, e.g. setting up a conda env with only Python+pip and using pip from there (or even creating virtualenvs), there isn't really much of a difference except that conda gets you a binary, and pyenv gets you compilation from source. It is technically possible for conda installation of Python packages to install packages without proper setuptools metadata, but this requires conda packages to be built incorrectly, and isn't typically an issue in practice.

The main advantage I see to use pyenv over conda to get Python itself is its much wider support of python versions and implementations (mainly pypy). But if we are whitelisting specific versions, I'm not sure we are benefitting much from that.

[yuvipanda]

At this point, I'm convinced that we don't have the maintainer resources required to support a non-conda and a conda based python (and that @minrk is pretty much always right, from the very beginning!). #190 has been stagnating, and you're right about the compilation problems.

I think a reasonable thing to do would be to:

1. Use conda to get python (only) in PythonBuildPack.
2. Base PipBuildPack on top of this, and install base / notebook from Pip
3. Base CondaBuildPack on top of PythonBuildPack, install base/notebook from pip.

I think this lets us only get python from conda, and not mix conda / pip libraries by default. How does this sound?

[yuvipanda]

Another option for solving the versions problem is to use https://launchpad.net/~deadsnakes/+archive/ubuntu/ppa after bionic arrives. But that too doesn't feel quite right, and a bit fragile.

[minrk]

I'll take a stab at your proposal to see how it looks. I think it's a good idea!

I, too, looked at deadsnakes and was disappointed that it didn't have the Pythons we wanted for our distro. It would certainly have been simplest. I had thought it was "all the Pythons for all the distros!"

In addition to pyenv's support for pypy, pipenv's support for pyenv might prove compelling. But if we can ensure that conda has installed the right Python (this isn't likely to include pypy any time soon), using conda for the base Python ought to still work.

If pyenv worked like homebrew and did public caches of the binary results of its builds, I would be 100% behind it. But I understand why it doesn't, with the infinite variety of platforms it supports. Every option we have seems to be almost great.

[yuvipanda]

w00t :) I think we should reconsider how we get pythons in the future if better options come along. but using conda seems simplest now.

…

On Thu, Feb 15, 2018 at 8:21 AM, Min RK ***@***.***> wrote: I'll take a stab at your proposal to see how it looks. I think it's a good idea! I, too, looked at deadsnakes and was disappointed that it didn't have the Pythons we wanted for our distro. It would certainly have been simplest. I had thought it was "all the Pythons for all the distros!" In addition to pyenv's support for pypy, pipenv's support for pyenv might prove compelling. But if we can ensure that conda has installed the right Python (this isn't likely to include pypy any time soon), using conda for the base Python ought to still work. If pyenv worked like homebrew and did public caches of the binary results of its builds, I would be 100% behind it. But I understand why it doesn't, with the infinite variety of platforms it supports. Every option we have seems to be *almost* great. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#185 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAB23v4JCoVdQn3Q3Bg2m_Mi4MBByXsXks5tVFlugaJpZM4RUn8h> .

-- Yuvi Panda T http://yuvi.in/blog

[minrk]

Yes, absolutely. Hopefully 'where Python comes from' shouldn't affect downstream users much, and we can try out new things as they come along.

If someone starts maintaining a "Python binaries for ubuntu + pyenv" archive, there's a good chance that will be ideal.