Support creating pyz files for different platforms

[dan-blanchard]

I need to add tests and docs for this, but I just wanted to post this before the weekend since I've actually gotten it working.

The key changes are:

1. Use pip download + wheel install instead of pip install for installing wheels. This way you can force installation into a directory even with incompatible wheels, and you can rely on pip download to get the correct wheels for a different platform.
2. Stop enforcing that --python exists, since you might be creating a .pyz file for a different machine.
3. Fix a bug in map_shared_objects where some internal .so files used by numpy would break the module_name splitting code.

If you want to try it out, just run:

shiv -p /path/to/python36/on/other/machine/python3 -o foo.pyz --python-version 36 --platform manylinux1_x86_64 --abi cp36m --implementation cp --only-binary=:all: Cython numpy

I didn't add any special args for cross-platform stuff, since we can rely on the pip download flags.

[coveralls]

Pull Request Test Coverage Report for Build 86

• 0 of 0 changed or added relevant lines in 0 files are covered.
• 21 unchanged lines in 3 files lost coverage.
• Overall coverage decreased (-1.8%) to 74.828%

---

Files with Coverage Reduction	New Missed Lines	%
.tox/py36/lib/python3.6/site-packages/shiv/builder.py	1	94.74%
.tox/py36/lib/python3.6/site-packages/shiv/pip.py	8	77.78%
.tox/py36/lib/python3.6/site-packages/shiv/cli.py	12	86.05%

Totals
Change from base Build 81:	-1.8%
Covered Lines:	217
Relevant Lines:	290

---

💛 - Coveralls

[lorencarvalho]

Hi @dan-blanchard !

Thanks for the PR, this looks pretty promising... Some thoughts:

Use pip download + wheel install instead of pip install for installing wheels. This way you can force installation into a directory even with incompatible wheels, and you can rely on pip download to get the correct wheels for a different platform.

This actually feels a little faster than the old pip install way, and I'm totally not opposed to switching to this instead of pip install, however, my first test revealed that something isn't quite right exclusively using wheels, I ran the following using your branch:

shiv -c hiss hiss-repl hiss-themes -o ~/bin/hiss

Hiss is a python repl based on ipython... It fails to bootstrap due to being unable to import some part of iPython.

Also, what about packages that don't publish wheels? We'll need some code to either build a wheel or fallback to pip install for this?

I've noticed while working on shiv that there are a number of packages that seem to require standard installation or they fail to work in mysterious ways. Which is why we decided to drop the notion of 'zip safety' in shiv. Too many packages expect things like __file__ to be real paths to reasonably support zip safety... which I'm bringing up because I think the shared object map is actually a relic from when we wanted to support zip safety (we had a custom Loader that extracted .so files before importing them, due to dlopen limitations).

Stop enforcing that --python exists, since you might be creating a .pyz file for a different machine.

I've kind of gone back and forth on how best to handle this, should we just use #!/usr/bin/env python3? Should we offer some combination of cli args to construct a shebang? What do you think? I'm ok with the little warning you are printing :)

Fix a bug in map_shared_objects where some internal .so files used by numpy would break the module_name splitting code.

Yeah we can just remove all the shared object map stuff :)

[lorencarvalho]

aha, so I figured out why hiss wasn't working, seems that if there is no wheel then pip download grabs the tarball (duh), so we just need to sort out what to do with those types of packages.

Collecting backcall
  Using cached https://files.pythonhosted.org/packages/84/71/c8ca4f5bb1e08401b916c68003acf0a0655df935d74d93bf3f3364b310e0/backcall-0.1.0.tar.gz

[dan-blanchard]

Ah, I was just testing the all-binary packages model. I’ll add a step that builds the tarballs into wheels.

…

On Fri, Apr 27, 2018 at 8:53 PM Loren Carvalho ***@***.***> wrote: aha, so I figured out why hiss wasn't working, seems that if there is no wheel then pip download grabs the tarball (duh), so we just need to sort out what to do with those types of packages. Collecting backcall Using cached https://files.pythonhosted.org/packages/84/71/c8ca4f5bb1e08401b916c68003acf0a0655df935d74d93bf3f3364b310e0/backcall-0.1.0.tar.gz — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#7 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA7l2cHVHMVjGl-njvyfQDl4juteVMaPks5ts72NgaJpZM4TrAly> .

[lorencarvalho]

@dan-blanchard

Perhaps instead of replacing our use of pip install we should consider the pip wheel command.. It offers the same Index options as install (such as --find-links, --no-index, etc) which are vital to our internal use, while also fetching wheels and building them when they are not already available.

Then, to build a multi-platform pyz, you'd simply need to download the wheels using pip itself, via the download args you gave as an example ( --python-version 36 --platform manylinux1_x86_64 --abi cp36m --implementation cp --only-binary=:all: Cython numpy ), and a 2nd step (building the pyz via shiv) using -f /path/to/wheelhouse --no-index

Sure, it's an extra step, which can complicate build scripts (but only slightly), but it allows us to retain the important behavior of packaging a single site-packages directory (vital for pkg_resources performance) while also preserving the original API.

I am still concerned about .pth files being missing from packages that are installed via wheel. Haven't had a chance to dig into that or investigate how nspkg.pth files are handled.

-- Loren

[dan-blanchard]

Yeah, pip wheel was what I was thinking we could use to create the wheels when we’ve got source packages and you’re not specifying all-binary.

…

On Mon, Apr 30, 2018 at 12:38 AM Loren Carvalho ***@***.***> wrote: @dan-blanchard <https://github.com/dan-blanchard> Perhaps instead of replacing our use of pip install we should consider the pip wheel command.. It offers the same Index options as install (such as --find-links, --no-index, etc) which are vital to our internal use, while also fetching wheels and building them when they are not already available. Then, to build a multi-platform pyz, you'd simply need to *download* the wheels using pip itself, via the download args you gave as an example ( --python-version 36 --platform manylinux1_x86_64 --abi cp36m --implementation cp --only-binary=:all: Cython numpy ), and a 2nd step (building the pyz via shiv) using -f /path/to/wheelhouse --no-index Sure, it's an extra step, which can complicate build scripts (but only slightly), but it allows us to retain the important behavior of packaging a single site-packages directory (vital for pkg_resources performance) while also preserving the original API. I am still concerned about .pth files being missing from packages that are installed via wheel. Haven't had a chance to dig into that or investigate how nspkg.pth files are handled. -- Loren — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#7 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA7l2fqmiFoyQBZF-4rZnmsWENGYvhw6ks5ttpVCgaJpZM4TrAly> .

[lorencarvalho]

So, I think I found a "workaround" that would allow you to accomplish what you want (manylinux compatible .pyz files made with shiv):

Step 1: download the desired wheels from pypi

lcarvalh-mn1 ~/cross-plat-shiv ❱❱❱ python3 -m pip download --python-version 36 --platform manylinux1_x86_64 --abi cp36m --implementation cp --only-binary=:all: Cython numpy
Collecting Cython
  Using cached https://files.pythonhosted.org/packages/19/eb/c4d9f3beafd5ac0615936860bcee41d93ca58f8734a16715da0037d2c468/Cython-0.28.2-cp36-cp36m-manylinux1_x86_64.whl
  Saved ./Cython-0.28.2-cp36-cp36m-manylinux1_x86_64.whl
Collecting numpy
  Using cached https://files.pythonhosted.org/packages/71/90/ca61e203e0080a8cef7ac21eca199829fa8d997f7c4da3e985b49d0a107d/numpy-1.14.3-cp36-cp36m-manylinux1_x86_64.whl
  Saved ./numpy-1.14.3-cp36-cp36m-manylinux1_x86_64.whl
Successfully downloaded Cython numpy

Step 2: rename them such that they pass pip's pep425tags check (yes, this is a bit of a hack)

lcarvalh-mn1 ~/cross-plat-shiv ❱❱❱ mv Cython-0.28.2-cp36-cp36m-manylinux1_x86_64.whl Cython-0.28.2-py3-none-any.whl
lcarvalh-mn1 ~/cross-plat-shiv ❱❱❱ mv numpy-1.14.3-cp36-cp36m-manylinux1_x86_64.whl numpy-1.14.3-py3-none-any.whl

Step 3: create a pyz with shiv

lcarvalh-mn1 ~/cross-plat-shiv ❱❱❱ shiv -o test.pyz -f . numpy-1.14.3-py3-none-any.whl Cython-0.28.2-py3-none-any.whl
 shiv! 🔪
Processing ./numpy-1.14.3-py3-none-any.whl
Processing ./Cython-0.28.2-py3-none-any.whl
Installing collected packages: numpy, Cython
Successfully installed Cython-0.28.2 numpy-1.14.3
 done

It won't work on Darwin, obviously (due to using manylinux wheels) but they DO seem to work as expected on Linux!

lcarvalh-mn1 ~/cross-plat-shiv ❱❱❱ scp test.pyz lcarvalh-ld1:~/test.pyz
test.pyz   100%   15MB   1.2MB/s   00:12
lcarvalh-mn1 ~/cross-plat-shiv ❱❱❱ ssh lcarvalh-ld1
Last login: Mon Apr 30 06:59:01 2018 from 2620:119:5042:900::12c
lcarvalh-ld1 ~ ❱❱❱ ./test.pyz
Python 3.6.1 (default, Apr 19 2017, 21:58:41)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-4)] on linux
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> # will work! on linux!
>>> import Cython
>>> import numpy
>>>
now exiting InteractiveConsole...

This method didn't require any changes to current shiv except fixing that map_shared_import bug you pointed out!

This leads me to believe that the way we ought to solve this permanently is by requesting/proposing/submitting a change to pip that would allow a --force option to pip install. Or maybe something more specific like --ignore-unsupported-wheels or some such. What do you think?

[lorencarvalho]

https://groups.google.com/forum/#!topic/pypa-dev/R4M8voZjW5Y

[warsaw]

I followed the twisty trail to the PyPA GH issue and followed up there, re skipping the PEP 425 checks.

[anthrotype]

Perhaps instead of replacing our use of pip install we should consider the pip wheel command.. It offers the same Index options as install [...], while also fetching wheels and building them when they are not already available

i'm probably stating the obvious, but I believe you can't cross-compile a sdist into a wheel. You'd have to build your pyz on the same platform which you are targeting. But in this case, pip install should already do the right thing (download wheel if exists or build one).

[warsaw]

I think it's theoretically possible to do some cross-compiling, but I'm not sure it's worth it, and even so it wouldn't be shiv's responsibility.

[dan-blanchard]

@sixninetynine I was out of town last week, so I missed a lot of the conversation here, and I'm not sure I'm fully caught up.

Perhaps instead of replacing our use of pip install we should consider the pip wheel command.. It offers the same Index options as install (such as --find-links, --no-index, etc) which are vital to our internal use, while also fetching wheels and building them when they are not already available.

Adding a pip wheel step for source packages makes lots of sense, and if we continue using wheel to install things instead of pip install, then them not letting us force installation before zipping is a non-issue.

pip wheel unfortunately does not let you specify different platforms like download does, so I think we would need both to make this easy for most users. I don't think we should make people download wheels separately, because that's kind of a pain (and a very common operation).

Step 2: rename them such that they pass pip's pep425tags check (yes, this is a bit of a hack)

That's what PEX's maintainers were recommending people do as a workaround instead of adding proper manylinux support, so... I'd rather not go that route.

[lorencarvalho]

Hiya @dan-blanchard

Yeah the conversation has definitely sprawled a bit 😄, we had some discussion on the pip tracker regarding overriding the pep425 checks as well, which imo would be the ideal way to solve it (as we could happily continue to rely on pip install behavior). However, from discussion on their tracker it seems that it would break a lot of assumptions in unexpected ways and potentially cause a lot of confusion for users (and headache for maintainers!). I totally respect their position, as ours is a pretty slim use case.

So let's define some scope to the desired outcome of this PR,

If your origin (build) machine is linux then creating manylinux compatible pyz's happens automatically just by passing --only-binary :all: (which is delegated to pip install)

❱❱❱ shiv -o test.pyz --only-binary :all: Cython numpy
 shiv! 🔪
Collecting Cython
  Using cached https://files.pythonhosted.org/packages/19/eb/c4d9f3beafd5ac0615936860bcee41d93ca58f8734a16715da0037d2c468/Cython-0.28.2-cp36-cp36m-manylinux1_x86_64.whl
Collecting numpy
  Using cached https://files.pythonhosted.org/packages/71/90/ca61e203e0080a8cef7ac21eca199829fa8d997f7c4da3e985b49d0a107d/numpy-1.14.3-cp36-cp36m-manylinux1_x86_64.whl
Installing collected packages: Cython, numpy
Successfully installed Cython-0.28.2 numpy-1.14.3
 done

Given that, what remaining gaps are we looking to fill? If we want to create manylinux pyz's from, say, OSX... that could be problematic given the current lay of the land. It would almost certainly mean we'd have to stop relying on pip install and move to using wheel or some combination of pip download and wheel.

The reason I'm somewhat resistant to that idea is that it would impose some reworking of our internal build plugins. We use https://github.com/linkedin/pygradle/ which pre-fetches all our requirements/deps based on gradle's dependency graph, so we already have a wheelhouse at shiv-creation time and can simply run shiv with --no-index --find-links /path/to/wheelhouse. Using download as our backend means we'd need to re-think that (and possibly other) workflows. Depending on pip install makes the workflow (and the code!) super simple.

[dan-blanchard]

We use https://github.com/linkedin/pygradle/ which pre-fetches all our requirements/deps based on gradle's dependency graph, so we already have a wheelhouse at shiv-creation time and can simply run shiv with --no-index --find-links /path/to/wheelhouse.

pip download supports --no-index and --find-links just like pip install. The last time I dug around in pip's internals I found that download is basically the same thing install uses before doing the actual installing.

[anthrotype]

If we want to create manylinux pyz's from, say, OSX... that could be problematic

you'd need to have Docker and pull the PyPA manylinux images and do the building inside it. You're basically running Linux on a Mac.

pip wheel unfortunately does not let you specify different platforms like download does

because you can't build for a different platform, at least not with distutils/setuptools.

[dan-blanchard]

@sixninetynine I've tested this changes both with and without specifying --python-implementation and the other cross-platform pip args, and it seems to work just fine. The arg filtering is a little ugly, but annoyingly pip download takes some args that pip wheel chokes on.

Also, I cannot figure out why mypy can't find the wheel module. I've never used mypy before. All the tests pass though.

[dan-blanchard]

I actually simplified the wheel building step a bit by making it only specifically try to make wheels out of packages that wear downloaded as sdists, so it doesn't have to do any of that ugly filtering anymore.

[dan-blanchard]

I've resolved all the conflicts on this branch. @sixninetynine is there anything else you'd like to see here?

[dan-blanchard]

Ugh, apparently there are real test failures now. I thought it was only the mypy thing again. I'll look into it.

[lorencarvalho]

hey @dan-blanchard

Right now I am hoping that pypa/pip#5404 is accepted/merged, which (combined with #34) ought to effectively implement this feature.

[dan-blanchard]

Oh cool, I hadn't seen your PR to add options to install. That would be amazing if it got accepted.

[lorencarvalho]

hey @dan-blanchard

just a heads up, my PR landed in pip.. if you upgrade your local pip (shiv lists pip as a dependency, but ensuring a specific version of pip is currently out of shiv's purview) then this functionality is supported when using bleeding edge pip. Once 18.1 is released it'll be more readily available:

step 1: ensure you have latest bleeding edge pip

python3 -m pip install --upgrade git+git://github.com/pypa/pip.git#egg=pip

step 2: create a manylinux pyz (on osx!)

darwin ~ $ shiv -o foo.pyz -p  "/usr/bin/env python3" --python-version 36 --platform manylinux1_x86_64 --abi cp36m --implementation cp --only-binary=:all: Cython numpy
 shiv! 🔪
Collecting Cython
  Using cached https://files.pythonhosted.org/packages/19/8e/32b280abb0947a96cdbb8329fb2014851a21fc1d099009f946ea8a8202c3/Cython-0.28.5-cp36-cp36m-manylinux1_x86_64.whl
Collecting numpy
  Using cached https://files.pythonhosted.org/packages/88/29/f4c845648ed23264e986cdc5fbab5f8eace1be5e62144ef69ccc7189461d/numpy-1.15.0-cp36-cp36m-manylinux1_x86_64.whl
Installing collected packages: Cython, numpy
Successfully installed Cython-0.28.5 numpy-1.15.0
 done

step 3: ship it over to a linux box

darwin ~ $ scp foo.pyz lcarvalh-ld1:~/
foo.pyz                                                                                              100%   17MB   8.9MB/s   00:01

step 4: now run it (on linux)

linux ~ $ ./foo.pyz
Python 3.6.1 (default, Apr 19 2017, 21:58:41)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-4)] on linux
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> import numpy
>>> import Cython
>>

[dan-blanchard]

@sixninetynine Awesome! Thanks for the tip. I'll close this PR since it's no longer necessary.