Observations on pypy2.7 and interoperability with existing packages

[mariusvniekerk]

Using this environment
conda create -n pypy -c conda-forge python=2.7 pypy2.7 ipython

Pure python packages

By just exporting a $PYTHONPATH or doing some symlinking magic I can get IPython to work from the conda installed package. By default pypy looks for (and installs) python packages in $PREFIX/site-packages, unlike $PREFIX/lib/python2.7/site-packages.

Attempting to use a conda-provided numpy fails.

Pypy will only import shared objects if they have the following structure

{modulename}.pypy-41.so

These are created when pypy -m pip instal numpy is run for example.

Mitigations

• For python packages that have python c extension modules, those need to be recompiled
• For pure python packages we can provide a post-install script that adds in the symlinks required

[mariusvniekerk]

cc @jakirkham

[jakirkham]

I wonder if we could fix the PyPy build to look in $PREFIX/lib/python2.7/site-packages. That would save a lot of work in downstream packages.

[mariusvniekerk]

So we can, although the issue is if there is some normal .so modules in there, they won't work

[jakirkham]

Sure. Though it sounds like given PyPy's naming scheme there really isn't any risk of conflict.

[mariusvniekerk]

Biggest pain would just be to change all recipes to depend on this (and the py3.5 variant) package.

Wonder if there is a way in conda to treat this as providing a python 2.7. Similar to Debian's alternatives.

Though I guess with some of the conda build 3 pieces we can get that with some template variables?

[mariusvniekerk]

@msarahan thoughts?

[msarahan]

You'll need some kind of mutual exclusivity scheme. Having package names the same is a convenient way, but may not be an option here. We had to deal with this recently with R and microsoft's MRO. More info at conda-forge/r-base-feedstock#34

My advice would be to definitely not try to use "features" for this.

[mariusvniekerk]

yeah my experience with features makes me very wary

[mariusvniekerk]

would something like making pypy just produce a python with a weird version work?

[msarahan]

It might, but I'd expect lots of very strange corner cases. Python in particular is special-cased in a lot of different ways across conda and conda-build, and it would take a lot of experimentation to see all the different quirks and iron them out.

[mariusvniekerk]

i guess the better model would be that

requirements:
   build: 
      - python

becomes

requirements:
   build:
      - {{ python_variant }}

and have conda-build 3 render recipes appropriately with pypy and python as variants. Feels less hacky at least

[msarahan]

yes, that seems like it might work. That will make looping over python versions harder, but you could mitigate that with explicit jinja vars for it:

requirements:
   build:
      - {{ python_variant }}  {{ python }}

This is necessary because right now, it's depending on matching the package name (in the absence of any version constraint) with a corresponding variant key. If the value of python_variant isn't the same as your python version matrix, you'll need to be explicit this way.

[mariusvniekerk]

yeah, essentially making the python version matrix something like

- python_variant: pypy
  python: 2.7
- python_variant: python
  python: 2.7

i assume we can still do things like skipping if the variant is pypy. this also technically opens up something like doing the same for ironpython / jython

[jakirkham]

We would need a pypy metapackage to make that work though.

[mariusvniekerk]

Certainly, I'll see if I can build a minimal numeric stack for pypy internally at my company with conda build 3.

Guess the main one to figure out is how to keep cpython out of the environment / how to score packages built against pypy higher at resolution time for an env already containing pypy.

I'm assuming modern conda can do this without resorting to channel priority.

[leycec]

Necrobump.

PyPy is rapidly approaching CPython parity.

With respect to worst-case performance, package compatibility, stdlib compliance, and API conformance, PyPy is nearly there and getting closer with every major release. Genuine PyPy support from conda-forge would lend an essential "shot in the arm" to the geriatric CPython ecosystem. While the reference implementations of both Ruby and R now bundle out-of-the-box JITs, CPython's development team and user community remain mistakenly hostile to the topic.

In theory, conda can solve this. In practice, conda-build appears to lack proper support for Debian-style alternatives or Gentoo-style virtuals (i.e., metapackages whose sole raison d'être is to proxy the set of all other packages satisfying a common API), which... seems less than ideal.

Nonetheless, @msarahan noted a deceptively simple (but possibly horrifying) solution. When in doubt, do what the r channel does. Specifically, r is cautiously adding support for both the reference R implementation and Microsoft R Open (MRO) by defining a new _r-mutex metapackage guaranteeing mutual exclusivity between those two implementations by... wait for it! Tracking a feature. </elongated_sigh>

Yes. Tracking a feature. Which appears to be the poor man's Gentoo USE flag, only sort-of broken in conda-build 2. ^{I take it that's bad.}

Will the conda-forge/r-base-feedstock#34 solution suffice to reliably solve the CPython <-> PyPy divide? Probably not, because this is the real world. Will conda-build 3 resolve all outstanding issues surrounding features, for it is our prophesied vaporware messiah? Probably not, because conda-build 3 will only land after the Great Purifying Fire of 2022. At which point, I'm probably a mutant trainwreck mucking about in the cesium-laden dirt for that last remaining worm rat.

Still, working towards any solution – even a non-ideal solution tracking a broken feature like a crippled mutant-baby through the wasteland – might be preferable to no solution. That's what we currently have. We can do better than nothing, right?

Anecdotal use case: our kinda PyPy-compatible multiphysics biology simulator would 😍 to officially support PyPy. Technically, we could do so now. In practice, the lack of support for PyPy in conda-forge is a "hard blocker" to us even bothering. The OpenBLAS-optimized NumPy stack hosted by conda-forge is currently critical to our workflow. With every increasingly optimized PyPy release, however, that calculus begins to shift in favour of PyPy.

Maybe someday we'll be able to have our OpenBLAS-optimized NumPy and PyPy it too. Until then, this necrobump is for you, conda-forge.

[astrojuanlu]

Despite the slightly snarky (but nevertheless entertaining) tone of the message above, I think there are also some valid points and I'm also very interested in seeing better PyPy support in conda (or better yet, PyPy3 - I only care about Python 3).

In the meanwhile, I sent a message to scipy-dev about having wheels in PyPy3, to see if there's some interest.

[rgommers]

(or better yet, PyPy3 - I only care about Python 3).

NumPy 1.16 (end 2018 or early 2019) will be the last release to support 2.7, after that 2.7 is effectively dead for the scientific stack. So yes, don't bother putting effort into that - PyPy 3 only would be better.

[astrojuanlu]

Is this still a problem with the ongoing conda-forge effort to provide PyPy packages?

[jakirkham]

No, we dropped Python 2. So this can be closed as “won’t fix”.