Add support for installation of binary distutils packages on Windows #520

Closed
technomalogical opened this Issue May 1, 2012 · 20 comments

Comments

Projects
None yet
8 participants
@technomalogical

@ianb has said in the past that pip could eventually support binary packages on Windows:

I've opened this issue to track discussion about how this could work.

@technomalogical

This comment has been minimized.

Show comment
Hide comment
@technomalogical

technomalogical May 1, 2012

Based on my own research, it appears that easy_install treats distutils .exe files as zip packages. I'm willing to do the work necessary, but need to find out more about how pip deals with PyPI and local packages. I believe there is a lack of metadata about binary packages in PyPI for a particular platform. Can anyone comment?

Based on my own research, it appears that easy_install treats distutils .exe files as zip packages. I'm willing to do the work necessary, but need to find out more about how pip deals with PyPI and local packages. I believe there is a lack of metadata about binary packages in PyPI for a particular platform. Can anyone comment?

@ianb

This comment has been minimized.

Show comment
Hide comment
@ianb

ianb May 1, 2012

Contributor

The metadata in PyPI is all embedded directly in the egg's name. PackageFinder could potentially parse out that metadata.

Contributor

ianb commented May 1, 2012

The metadata in PyPI is all embedded directly in the egg's name. PackageFinder could potentially parse out that metadata.

@rcludwick

This comment has been minimized.

Show comment
Hide comment
@rcludwick

rcludwick May 9, 2012

I would love to see this work in Linux as well as I am dependent upon installing binary eggs.

I would love to see this work in Linux as well as I am dependent upon installing binary eggs.

@corydolphin

This comment has been minimized.

Show comment
Hide comment
@corydolphin

corydolphin Jan 24, 2013

If anyone else is interested in working with me to make this happen, please let me know. I have started to take notes on the changes necessary. My vision is to add a new flag to the install command i.e. --winbin to represent a user wishes to use windows binaries on pypi

Currently, I know the following changes need to be made:

  • In pip.index.PackageFinder does not include .exes in its search for links
  • In pip.comands.install must support the flag, and perform proper checks for Windows
  • In pip.req needs to implement unzipping for an .exe archive.

Does anyone with more knowledge of pip have interest in helping with the changes? I am not familiar with the codebase, and would love the changes to be as robust and elegant as possible. Currently, I simply use easy_install on Windows binaries hosted on pypi, for packages which require compilation on Windows.

If anyone else is interested in working with me to make this happen, please let me know. I have started to take notes on the changes necessary. My vision is to add a new flag to the install command i.e. --winbin to represent a user wishes to use windows binaries on pypi

Currently, I know the following changes need to be made:

  • In pip.index.PackageFinder does not include .exes in its search for links
  • In pip.comands.install must support the flag, and perform proper checks for Windows
  • In pip.req needs to implement unzipping for an .exe archive.

Does anyone with more knowledge of pip have interest in helping with the changes? I am not familiar with the codebase, and would love the changes to be as robust and elegant as possible. Currently, I simply use easy_install on Windows binaries hosted on pypi, for packages which require compilation on Windows.

@pfmoore

This comment has been minimized.

Show comment
Hide comment
@pfmoore

pfmoore Jan 24, 2013

Member

In my view, wheels are a better solution here. The wheel support for pip is available now in a separate branch, due to be committed once the relevant standards PEPs are accepted. And converting wininst installers to wheels is possible using the wheel package.

The (relatively minor, admittedly) problem with direct wininst installation (at least as provided by easy_install) is that wininsts get installed in egg directories, rather than in "flat" form, which causes sys.path to become unnecessarily long.

Member

pfmoore commented Jan 24, 2013

In my view, wheels are a better solution here. The wheel support for pip is available now in a separate branch, due to be committed once the relevant standards PEPs are accepted. And converting wininst installers to wheels is possible using the wheel package.

The (relatively minor, admittedly) problem with direct wininst installation (at least as provided by easy_install) is that wininsts get installed in egg directories, rather than in "flat" form, which causes sys.path to become unnecessarily long.

@blink1073

This comment has been minimized.

Show comment
Hide comment
@blink1073

blink1073 Jan 26, 2013

Why not let the exe installer run as-is, rather than dissecting it, at least as a start.

Why not let the exe installer run as-is, rather than dissecting it, at least as a start.

@blink1073

This comment has been minimized.

Show comment
Hide comment
@blink1073

blink1073 Jan 26, 2013

Played a little:

import zipfile
zf = zipfile.Zipfile('c:\\path\\to\\binary')
zf.extractall('c:\\temp\install\\path')
zf.close()

This creates a folder named PLATLIB or PURELIB, and a SCRIPTS folder, if applicable. The contents of these folders can be copied as-is to the site-packages and scripts folders. I used the numpy installer from pypi and Christoph Gohlke's (@cgohlke) scikit-image on my 32-bit Python 2.7, and ran the scikit-image test suite successfully (which relies on numpy).

Played a little:

import zipfile
zf = zipfile.Zipfile('c:\\path\\to\\binary')
zf.extractall('c:\\temp\install\\path')
zf.close()

This creates a folder named PLATLIB or PURELIB, and a SCRIPTS folder, if applicable. The contents of these folders can be copied as-is to the site-packages and scripts folders. I used the numpy installer from pypi and Christoph Gohlke's (@cgohlke) scikit-image on my 32-bit Python 2.7, and ran the scikit-image test suite successfully (which relies on numpy).

@corydolphin

This comment has been minimized.

Show comment
Hide comment
@corydolphin

corydolphin Jan 26, 2013

Because I was unable to find documentation on options the bdist_wininst binary accepts, and thus could not find a way to control where the package is installed, making it useless in a virtualenv. Extracting the binary creates folders as you describe, and should be simply copied to sites-packages, as far as I can tell.

On Saturday, January 26, 2013, Steven Silvester wrote:

Why not let the exe installer run as-is, rather than dissecting it, at
least as a start.


Reply to this email directly or view it on GitHubhttps://github.com/pypa/pip/issues/520#issuecomment-12735883.

Because I was unable to find documentation on options the bdist_wininst binary accepts, and thus could not find a way to control where the package is installed, making it useless in a virtualenv. Extracting the binary creates folders as you describe, and should be simply copied to sites-packages, as far as I can tell.

On Saturday, January 26, 2013, Steven Silvester wrote:

Why not let the exe installer run as-is, rather than dissecting it, at
least as a start.


Reply to this email directly or view it on GitHubhttps://github.com/pypa/pip/issues/520#issuecomment-12735883.

@blink1073

This comment has been minimized.

Show comment
Hide comment
@blink1073

blink1073 Jan 26, 2013

The same would apply to a virtualenv, copying the extracted source folder directly. I copied the extracted numpy folder to a new virtualenv and did the following successfully:

import numpy
numpy.test()

The same would apply to a virtualenv, copying the extracted source folder directly. I copied the extracted numpy folder to a new virtualenv and did the following successfully:

import numpy
numpy.test()
@blink1073

This comment has been minimized.

Show comment
Hide comment
@blink1073

blink1073 Jan 26, 2013

So, all that is left is implementing this behavior in pip. Does anyone else think this should be the default on Windows, searching for an exe and using that rather than compiling it, which is fraught with peril for new users?

So, all that is left is implementing this behavior in pip. Does anyone else think this should be the default on Windows, searching for an exe and using that rather than compiling it, which is fraught with peril for new users?

@blink1073

This comment has been minimized.

Show comment
Hide comment
@blink1073

blink1073 Jan 26, 2013

I created a proof of concept that grabs the appropriate windows binary from pypi if it exists, extracts it, and copies the source to the appropriate directories, accommodating virtualenvs. You can provide a file name if they do not use a standard name, such as ipython.

I created a proof of concept that grabs the appropriate windows binary from pypi if it exists, extracts it, and copies the source to the appropriate directories, accommodating virtualenvs. You can provide a file name if they do not use a standard name, such as ipython.

@pfmoore

This comment has been minimized.

Show comment
Hide comment
@pfmoore

pfmoore Jan 28, 2013

Member

Some suggestions that you should consider if you want this to be added to pip:

  • How many packages on PyPI supply wininst installers? The last time I looked, a lot of packages (that I was interested in) didn't provide wininst executables (for example, pywin32), or only provided them for older releases (for example, lxml). A query on the PyPI index would be interesting here.
  • What should the behaviour be where a project provides binaries for older versions, but not for the latest? Some projects provide wininst installers for pure-python packages, so saying "use the latest binary" might be wrong.
  • What if the user does have a compiler? Should pip ignore binaries? Probably yes for things that have no dependencies and can be built easily, but probably no if the project has build dependencies that are non-trivial to set up.
  • Numpy is an interesting example - the PyPI download is not the same as the "official" (superpack) binaries from the numpy home site. The official binaries have a more complex structure (3 subtly-different architecture-dependent versions, chosen at install time) which is still extractable by the unzip approach, but needs extra work to do so. If pip supports installing wininst installers from PyPI, people will end up using (presumably) suboptimal builds by default, which may not be a win overall.

It might be worth making your proof of concept into a standalone package that people can install. Publicise it and see how many people use it (I might well do so, for example). If it gets a lot of use, that would be a strong argument for inclusion in pip. And until that happens, the standalone utility may be enough for people who need the functionality.

Member

pfmoore commented Jan 28, 2013

Some suggestions that you should consider if you want this to be added to pip:

  • How many packages on PyPI supply wininst installers? The last time I looked, a lot of packages (that I was interested in) didn't provide wininst executables (for example, pywin32), or only provided them for older releases (for example, lxml). A query on the PyPI index would be interesting here.
  • What should the behaviour be where a project provides binaries for older versions, but not for the latest? Some projects provide wininst installers for pure-python packages, so saying "use the latest binary" might be wrong.
  • What if the user does have a compiler? Should pip ignore binaries? Probably yes for things that have no dependencies and can be built easily, but probably no if the project has build dependencies that are non-trivial to set up.
  • Numpy is an interesting example - the PyPI download is not the same as the "official" (superpack) binaries from the numpy home site. The official binaries have a more complex structure (3 subtly-different architecture-dependent versions, chosen at install time) which is still extractable by the unzip approach, but needs extra work to do so. If pip supports installing wininst installers from PyPI, people will end up using (presumably) suboptimal builds by default, which may not be a win overall.

It might be worth making your proof of concept into a standalone package that people can install. Publicise it and see how many people use it (I might well do so, for example). If it gets a lot of use, that would be a strong argument for inclusion in pip. And until that happens, the standalone utility may be enough for people who need the functionality.

@blink1073

This comment has been minimized.

Show comment
Hide comment
@blink1073

blink1073 Jan 28, 2013

I see what you mean, and that would argue for the use of a command flag. I will make a standalone package, and add the option to install from a local file. This opens up the use of Christoph Gohlke's files (or Numpy's superpack), with the added benefit of being installable in a virtualenv.

I see what you mean, and that would argue for the use of a command flag. I will make a standalone package, and add the option to install from a local file. This opens up the use of Christoph Gohlke's files (or Numpy's superpack), with the added benefit of being installable in a virtualenv.

@pfmoore

This comment has been minimized.

Show comment
Hide comment
@pfmoore

pfmoore Jan 28, 2013

Member

One problem with a command flag is that you may want different answers for different projects, all of which may get picked up in a dependency scan. Whether this is an issue depends on how complicated you get of course (were you planning on integrating this with dependency scanning?)

Member

pfmoore commented Jan 28, 2013

One problem with a command flag is that you may want different answers for different projects, all of which may get picked up in a dependency scan. Whether this is an issue depends on how complicated you get of course (were you planning on integrating this with dependency scanning?)

@blink1073

This comment has been minimized.

Show comment
Hide comment
@blink1073

blink1073 Jan 28, 2013

Yes, ideally; baby steps my friend...

Yes, ideally; baby steps my friend...

@dholth

This comment has been minimized.

Show comment
Hide comment
@dholth

dholth Jan 28, 2013

Member

Are you familiar with wheel's wininst2wheel utility and the pip patch to install wheels?

On Jan 28, 2013, at 2:43 PM, Steven Silvester notifications@github.com wrote:

Yes, ideally; baby steps my friend...


Reply to this email directly or view it on GitHub.

Member

dholth commented Jan 28, 2013

Are you familiar with wheel's wininst2wheel utility and the pip patch to install wheels?

On Jan 28, 2013, at 2:43 PM, Steven Silvester notifications@github.com wrote:

Yes, ideally; baby steps my friend...


Reply to this email directly or view it on GitHub.

@blink1073

This comment has been minimized.

Show comment
Hide comment
@blink1073

blink1073 Jan 29, 2013

No; that does look like the best long term solution. I was hoping to offer some help to the scientific community now. I will pursue a standalone script for now, and leave the rest to you, @dholth. When the PEP is accepted, I will write a workflow that helps developers create Windows wheels using Wine, similar to this. Sound like a plan?

No; that does look like the best long term solution. I was hoping to offer some help to the scientific community now. I will pursue a standalone script for now, and leave the rest to you, @dholth. When the PEP is accepted, I will write a workflow that helps developers create Windows wheels using Wine, similar to this. Sound like a plan?

@pfmoore

This comment has been minimized.

Show comment
Hide comment
@pfmoore

pfmoore Jan 29, 2013

Member

Right now, I use wheels as follows:

  • Install the wheel package (one-off requirement)
  • Use wheel convert on a wininst
  • Use wheel install on the resulting wheel file

No need for a patched pip or anything, if you don't want it.

The downsides for me are:

  1. No automatic location and download of wininst files. This is not really a problem as nearly all of the wininst files I want are not hosted on PyPI anyway (they are usually from Christoph Gohlke's site)
  2. No dependency tracking (not really an issue given (1)).

That may or may not be a helpful workflow for you. Going forward, documentation of now to create wheels (which need not be just for Windows, btw!) would be excellent. Building on Christoph's site, it would also be good to look at how to better integrate 3rd-party builds of projects into the pip/PyPI infrastructure. Maybe simply having a separate site that presents the PyPI API but offers to host 3rd-party builds would be suitable. (But I'm getting way off topic now...)

Member

pfmoore commented Jan 29, 2013

Right now, I use wheels as follows:

  • Install the wheel package (one-off requirement)
  • Use wheel convert on a wininst
  • Use wheel install on the resulting wheel file

No need for a patched pip or anything, if you don't want it.

The downsides for me are:

  1. No automatic location and download of wininst files. This is not really a problem as nearly all of the wininst files I want are not hosted on PyPI anyway (they are usually from Christoph Gohlke's site)
  2. No dependency tracking (not really an issue given (1)).

That may or may not be a helpful workflow for you. Going forward, documentation of now to create wheels (which need not be just for Windows, btw!) would be excellent. Building on Christoph's site, it would also be good to look at how to better integrate 3rd-party builds of projects into the pip/PyPI infrastructure. Maybe simply having a separate site that presents the PyPI API but offers to host 3rd-party builds would be suitable. (But I'm getting way off topic now...)

@blink1073

This comment has been minimized.

Show comment
Hide comment
@blink1073

blink1073 Jan 29, 2013

Imagine a web service that can build wheels for all platforms, allowing developers to run one command and spit out all of the necessary files to pypi. Imagine a cross-platform GUI for pip where you choose your environment from a dropdown (including virtualenv), and manage your packages, seamlessly pulling wheels from pypi. Oh the glory...

Imagine a web service that can build wheels for all platforms, allowing developers to run one command and spit out all of the necessary files to pypi. Imagine a cross-platform GUI for pip where you choose your environment from a dropdown (including virtualenv), and manage your packages, seamlessly pulling wheels from pypi. Oh the glory...

@dstufft

This comment has been minimized.

Show comment
Hide comment
@dstufft

dstufft Jan 30, 2014

Member

I don't believe pip is going to be getting support for installing from wininsts. Since the time this ticket was active Wheel has been accepted and integrated into pip.

Member

dstufft commented Jan 30, 2014

I don't believe pip is going to be getting support for installing from wininsts. Since the time this ticket was active Wheel has been accepted and integrated into pip.

@dstufft dstufft closed this Jan 30, 2014

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment