Skip to content

Add support for installation of binary distutils packages on Windows #520

Closed
technomalogical opened this Issue May 1, 2012 · 20 comments

8 participants

@technomalogical

@ianb has said in the past that pip could eventually support binary packages on Windows:

I've opened this issue to track discussion about how this could work.

@technomalogical

Based on my own research, it appears that easy_install treats distutils .exe files as zip packages. I'm willing to do the work necessary, but need to find out more about how pip deals with PyPI and local packages. I believe there is a lack of metadata about binary packages in PyPI for a particular platform. Can anyone comment?

@ianb
ianb commented May 1, 2012

The metadata in PyPI is all embedded directly in the egg's name. PackageFinder could potentially parse out that metadata.

@rcludwick

I would love to see this work in Linux as well as I am dependent upon installing binary eggs.

@corydolphin

If anyone else is interested in working with me to make this happen, please let me know. I have started to take notes on the changes necessary. My vision is to add a new flag to the install command i.e. --winbin to represent a user wishes to use windows binaries on pypi

Currently, I know the following changes need to be made:

  • In pip.index.PackageFinder does not include .exes in its search for links
  • In pip.comands.install must support the flag, and perform proper checks for Windows
  • In pip.req needs to implement unzipping for an .exe archive.

Does anyone with more knowledge of pip have interest in helping with the changes? I am not familiar with the codebase, and would love the changes to be as robust and elegant as possible. Currently, I simply use easy_install on Windows binaries hosted on pypi, for packages which require compilation on Windows.

@pfmoore
Python Packaging Authority member
pfmoore commented Jan 24, 2013

In my view, wheels are a better solution here. The wheel support for pip is available now in a separate branch, due to be committed once the relevant standards PEPs are accepted. And converting wininst installers to wheels is possible using the wheel package.

The (relatively minor, admittedly) problem with direct wininst installation (at least as provided by easy_install) is that wininsts get installed in egg directories, rather than in "flat" form, which causes sys.path to become unnecessarily long.

@blink1073

Why not let the exe installer run as-is, rather than dissecting it, at least as a start.

@blink1073

Played a little:

import zipfile
zf = zipfile.Zipfile('c:\\path\\to\\binary')
zf.extractall('c:\\temp\install\\path')
zf.close()

This creates a folder named PLATLIB or PURELIB, and a SCRIPTS folder, if applicable. The contents of these folders can be copied as-is to the site-packages and scripts folders. I used the numpy installer from pypi and Christoph Gohlke's (@cgohlke) scikit-image on my 32-bit Python 2.7, and ran the scikit-image test suite successfully (which relies on numpy).

@corydolphin
@blink1073

The same would apply to a virtualenv, copying the extracted source folder directly. I copied the extracted numpy folder to a new virtualenv and did the following successfully:

import numpy
numpy.test()
@blink1073

So, all that is left is implementing this behavior in pip. Does anyone else think this should be the default on Windows, searching for an exe and using that rather than compiling it, which is fraught with peril for new users?

@blink1073

I created a proof of concept that grabs the appropriate windows binary from pypi if it exists, extracts it, and copies the source to the appropriate directories, accommodating virtualenvs. You can provide a file name if they do not use a standard name, such as ipython.

@pfmoore
Python Packaging Authority member
pfmoore commented Jan 28, 2013

Some suggestions that you should consider if you want this to be added to pip:

  • How many packages on PyPI supply wininst installers? The last time I looked, a lot of packages (that I was interested in) didn't provide wininst executables (for example, pywin32), or only provided them for older releases (for example, lxml). A query on the PyPI index would be interesting here.
  • What should the behaviour be where a project provides binaries for older versions, but not for the latest? Some projects provide wininst installers for pure-python packages, so saying "use the latest binary" might be wrong.
  • What if the user does have a compiler? Should pip ignore binaries? Probably yes for things that have no dependencies and can be built easily, but probably no if the project has build dependencies that are non-trivial to set up.
  • Numpy is an interesting example - the PyPI download is not the same as the "official" (superpack) binaries from the numpy home site. The official binaries have a more complex structure (3 subtly-different architecture-dependent versions, chosen at install time) which is still extractable by the unzip approach, but needs extra work to do so. If pip supports installing wininst installers from PyPI, people will end up using (presumably) suboptimal builds by default, which may not be a win overall.

It might be worth making your proof of concept into a standalone package that people can install. Publicise it and see how many people use it (I might well do so, for example). If it gets a lot of use, that would be a strong argument for inclusion in pip. And until that happens, the standalone utility may be enough for people who need the functionality.

@blink1073

I see what you mean, and that would argue for the use of a command flag. I will make a standalone package, and add the option to install from a local file. This opens up the use of Christoph Gohlke's files (or Numpy's superpack), with the added benefit of being installable in a virtualenv.

@pfmoore
Python Packaging Authority member
pfmoore commented Jan 28, 2013

One problem with a command flag is that you may want different answers for different projects, all of which may get picked up in a dependency scan. Whether this is an issue depends on how complicated you get of course (were you planning on integrating this with dependency scanning?)

@blink1073

Yes, ideally; baby steps my friend...

@dholth
Python Packaging Authority member
@blink1073

No; that does look like the best long term solution. I was hoping to offer some help to the scientific community now. I will pursue a standalone script for now, and leave the rest to you, @dholth. When the PEP is accepted, I will write a workflow that helps developers create Windows wheels using Wine, similar to this. Sound like a plan?

@pfmoore
Python Packaging Authority member
pfmoore commented Jan 29, 2013

Right now, I use wheels as follows:

  • Install the wheel package (one-off requirement)
  • Use wheel convert on a wininst
  • Use wheel install on the resulting wheel file

No need for a patched pip or anything, if you don't want it.

The downsides for me are:
1. No automatic location and download of wininst files. This is not really a problem as nearly all of the wininst files I want are not hosted on PyPI anyway (they are usually from Christoph Gohlke's site)
2. No dependency tracking (not really an issue given (1)).

That may or may not be a helpful workflow for you. Going forward, documentation of now to create wheels (which need not be just for Windows, btw!) would be excellent. Building on Christoph's site, it would also be good to look at how to better integrate 3rd-party builds of projects into the pip/PyPI infrastructure. Maybe simply having a separate site that presents the PyPI API but offers to host 3rd-party builds would be suitable. (But I'm getting way off topic now...)

@blink1073

Imagine a web service that can build wheels for all platforms, allowing developers to run one command and spit out all of the necessary files to pypi. Imagine a cross-platform GUI for pip where you choose your environment from a dropdown (including virtualenv), and manage your packages, seamlessly pulling wheels from pypi. Oh the glory...

@dstufft
Python Packaging Authority member
dstufft commented Jan 30, 2014

I don't believe pip is going to be getting support for installing from wininsts. Since the time this ticket was active Wheel has been accepted and integrated into pip.

@dstufft dstufft closed this Jan 30, 2014
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.