@ianb has said in the past that pip could eventually support binary packages on Windows:
I've opened this issue to track discussion about how this could work.
Based on my own research, it appears that easy_install treats distutils .exe files as zip packages. I'm willing to do the work necessary, but need to find out more about how pip deals with PyPI and local packages. I believe there is a lack of metadata about binary packages in PyPI for a particular platform. Can anyone comment?
The metadata in PyPI is all embedded directly in the egg's name. PackageFinder could potentially parse out that metadata.
I would love to see this work in Linux as well as I am dependent upon installing binary eggs.
If anyone else is interested in working with me to make this happen, please let me know. I have started to take notes on the changes necessary. My vision is to add a new flag to the install command i.e. --winbin to represent a user wishes to use windows binaries on pypi
Currently, I know the following changes need to be made:
Does anyone with more knowledge of pip have interest in helping with the changes? I am not familiar with the codebase, and would love the changes to be as robust and elegant as possible. Currently, I simply use easy_install on Windows binaries hosted on pypi, for packages which require compilation on Windows.
In my view, wheels are a better solution here. The wheel support for pip is available now in a separate branch, due to be committed once the relevant standards PEPs are accepted. And converting wininst installers to wheels is possible using the wheel package.
The (relatively minor, admittedly) problem with direct wininst installation (at least as provided by easy_install) is that wininsts get installed in egg directories, rather than in "flat" form, which causes sys.path to become unnecessarily long.
Why not let the exe installer run as-is, rather than dissecting it, at least as a start.
Played a little:
zf = zipfile.Zipfile('c:\\path\\to\\binary')
This creates a folder named PLATLIB or PURELIB, and a SCRIPTS folder, if applicable. The contents of these folders can be copied as-is to the site-packages and scripts folders. I used the numpy installer from pypi and Christoph Gohlke's (@cgohlke) scikit-image on my 32-bit Python 2.7, and ran the scikit-image test suite successfully (which relies on numpy).
The same would apply to a virtualenv, copying the extracted source folder directly. I copied the extracted numpy folder to a new virtualenv and did the following successfully:
So, all that is left is implementing this behavior in pip. Does anyone else think this should be the default on Windows, searching for an exe and using that rather than compiling it, which is fraught with peril for new users?
I created a proof of concept that grabs the appropriate windows binary from pypi if it exists, extracts it, and copies the source to the appropriate directories, accommodating virtualenvs. You can provide a file name if they do not use a standard name, such as ipython.
Some suggestions that you should consider if you want this to be added to pip:
It might be worth making your proof of concept into a standalone package that people can install. Publicise it and see how many people use it (I might well do so, for example). If it gets a lot of use, that would be a strong argument for inclusion in pip. And until that happens, the standalone utility may be enough for people who need the functionality.
I see what you mean, and that would argue for the use of a command flag. I will make a standalone package, and add the option to install from a local file. This opens up the use of Christoph Gohlke's files (or Numpy's superpack), with the added benefit of being installable in a virtualenv.
One problem with a command flag is that you may want different answers for different projects, all of which may get picked up in a dependency scan. Whether this is an issue depends on how complicated you get of course (were you planning on integrating this with dependency scanning?)
Yes, ideally; baby steps my friend...
No; that does look like the best long term solution. I was hoping to offer some help to the scientific community now. I will pursue a standalone script for now, and leave the rest to you, @dholth. When the PEP is accepted, I will write a workflow that helps developers create Windows wheels using Wine, similar to this. Sound like a plan?
Right now, I use wheels as follows:
No need for a patched pip or anything, if you don't want it.
The downsides for me are:
1. No automatic location and download of wininst files. This is not really a problem as nearly all of the wininst files I want are not hosted on PyPI anyway (they are usually from Christoph Gohlke's site)
2. No dependency tracking (not really an issue given (1)).
That may or may not be a helpful workflow for you. Going forward, documentation of now to create wheels (which need not be just for Windows, btw!) would be excellent. Building on Christoph's site, it would also be good to look at how to better integrate 3rd-party builds of projects into the pip/PyPI infrastructure. Maybe simply having a separate site that presents the PyPI API but offers to host 3rd-party builds would be suitable. (But I'm getting way off topic now...)
Imagine a web service that can build wheels for all platforms, allowing developers to run one command and spit out all of the necessary files to pypi. Imagine a cross-platform GUI for pip where you choose your environment from a dropdown (including virtualenv), and manage your packages, seamlessly pulling wheels from pypi. Oh the glory...
I don't believe pip is going to be getting support for installing from wininsts. Since the time this ticket was active Wheel has been accepted and integrated into pip.