New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Issue #2140: Build wheels during pip install #2618

Merged
merged 3 commits into from Apr 13, 2015

Conversation

Projects
None yet
@rbtcollins
Contributor

rbtcollins commented Mar 30, 2015

This branch is building up to tackle issue #2140 - build wheels during install and then install those wheels.

@rbtcollins

This comment has been minimized.

Show comment
Hide comment
@rbtcollins

rbtcollins Mar 30, 2015

Contributor

The topo sort thuings are from pull #2616

Contributor

rbtcollins commented Mar 30, 2015

The topo sort thuings are from pull #2616

@sigmavirus24

This comment has been minimized.

Show comment
Hide comment
@sigmavirus24

sigmavirus24 Mar 30, 2015

Member

This sounds exciting

Member

sigmavirus24 commented Mar 30, 2015

This sounds exciting

@alex

This comment has been minimized.

Show comment
Hide comment
@alex

alex Mar 31, 2015

Member

Can all the ensure_dir changes by refactored out into their own PR?

Member

alex commented Mar 31, 2015

Can all the ensure_dir changes by refactored out into their own PR?

@rbtcollins

This comment has been minimized.

Show comment
Hide comment
@rbtcollins

rbtcollins Mar 31, 2015

Contributor

They're a single commit. Can pull it out if desired.

Contributor

rbtcollins commented Mar 31, 2015

They're a single commit. Can pull it out if desired.

@rbtcollins

This comment has been minimized.

Show comment
Hide comment
@rbtcollins

rbtcollins Apr 1, 2015

Contributor

Its mainly tested via all the existing tests, but I added an explicit test looking at the output to be sure building was happening and handling of non-wheelable things.

Contributor

rbtcollins commented Apr 1, 2015

Its mainly tested via all the existing tests, but I added an explicit test looking at the output to be sure building was happening and handling of non-wheelable things.

@rbtcollins

This comment has been minimized.

Show comment
Hide comment
@rbtcollins

rbtcollins Apr 3, 2015

Contributor

I'm not sure why we're seeing those failures. Its not as simple as hashseed - I've tried using the same hashseed locally. It might be an isolation failure, or something. There's obviously an underlying issue, but until I can provoke it I can't fix it :(

I'm going to try adding diagnostic changes on top of the stack to do that.

Contributor

rbtcollins commented Apr 3, 2015

I'm not sure why we're seeing those failures. Its not as simple as hashseed - I've tried using the same hashseed locally. It might be an isolation failure, or something. There's obviously an underlying issue, but until I can provoke it I can't fix it :(

I'm going to try adding diagnostic changes on top of the stack to do that.

Show outdated Hide outdated pip/utils/__init__.py Outdated
@dstufft

This comment has been minimized.

Show comment
Hide comment
@dstufft

dstufft Apr 7, 2015

Member

This will need to be rebased (sorry!).

Member

dstufft commented Apr 7, 2015

This will need to be rebased (sorry!).

@dstufft dstufft referenced this pull request Apr 7, 2015

Closed

Build wheels #2649

@dstufft

This comment has been minimized.

Show comment
Hide comment
@dstufft

dstufft Apr 7, 2015

Member

For the record, I rebased this and submitted a new pull request for it (#2649). That is failing tests due to one of the things I noticed in the other PR.... However I'm not sure how it was passing tests previously. I don't see anything in the diff that has it ensuring that Wheel is installed...

Member

dstufft commented Apr 7, 2015

For the record, I rebased this and submitted a new pull request for it (#2649). That is failing tests due to one of the things I noticed in the other PR.... However I'm not sure how it was passing tests previously. I don't see anything in the diff that has it ensuring that Wheel is installed...

@dstufft

This comment has been minimized.

Show comment
Hide comment
@dstufft

dstufft Apr 7, 2015

Member

Bringing over my comments from #2649 I see the following issues with this currently:

  • It attempts to run bdist_wheel even whenever the wheel library is not installed. This leads to confusing output where it appears the installation failed. I think this PR should just not attempt to build wheels if the wheel library isn't installed, and then a future PR can add wheel (and setuptools) as an implicit setup_requires.
  • It downloads wheels and stores them inside of the wheel cache even if it already is a wheel on the index, ideally the wheel cache will only contain things which we actually need to build a wheel for and not ones which already have a wheel. We already will have the wheel file saved locally inside the http cache, so storing it here is just duplicating information.
  • The wheel cache just dumps all wheels it builds into a single directory, however this means that configured indexes "leak" into this, e.g. if i do pip install -i https://internal/simple/ foo where I have a forked copy of foo with a version of 1.0+myfoo, and then I execute in a fresh environment pip install foo, if 1.0 is the latest version on PyPI it'll find my 1.0+myfoo that was built from the other index. A possible solution to this is to simply add a per index wheel cache instead of a index agnostic wheel cache.
  • Even when things are working, the output isn't the greatest (see below).
  • It's building wheels for things like pip install ., which means that for instance, installing pip in that way puts a 7.0.0.dev0 wheel of pip into the wheel cache dir. A lot of people don't bump versions for development so it's possible you'll get a corrupted wheel cached. I think perhaps wheels should only be cached + built when coming from a repository of some form and not a direct link (either direct link to a tarball or a file path or what have you).
  • The --cache-dir option isn't controlling where the wheels are being cached too.
  • The --no-cache-dir seems to be somewhat respected, but in a weird way, it tells me it's building Wheels to put into destination None and ultimately the build fails and it installs from sdist.
$ pip install lxml
Collecting lxml
  File was already downloaded /Users/dstufft/Library/Caches/pip/wheels/lxml-3.4.2-cp34-cp34m-macosx_10_10_x86_64.whl
Skipping lxml, due to already being wheel.
Installing collected packages: lxml
Successfully installed lxml-3.4.2
Member

dstufft commented Apr 7, 2015

Bringing over my comments from #2649 I see the following issues with this currently:

  • It attempts to run bdist_wheel even whenever the wheel library is not installed. This leads to confusing output where it appears the installation failed. I think this PR should just not attempt to build wheels if the wheel library isn't installed, and then a future PR can add wheel (and setuptools) as an implicit setup_requires.
  • It downloads wheels and stores them inside of the wheel cache even if it already is a wheel on the index, ideally the wheel cache will only contain things which we actually need to build a wheel for and not ones which already have a wheel. We already will have the wheel file saved locally inside the http cache, so storing it here is just duplicating information.
  • The wheel cache just dumps all wheels it builds into a single directory, however this means that configured indexes "leak" into this, e.g. if i do pip install -i https://internal/simple/ foo where I have a forked copy of foo with a version of 1.0+myfoo, and then I execute in a fresh environment pip install foo, if 1.0 is the latest version on PyPI it'll find my 1.0+myfoo that was built from the other index. A possible solution to this is to simply add a per index wheel cache instead of a index agnostic wheel cache.
  • Even when things are working, the output isn't the greatest (see below).
  • It's building wheels for things like pip install ., which means that for instance, installing pip in that way puts a 7.0.0.dev0 wheel of pip into the wheel cache dir. A lot of people don't bump versions for development so it's possible you'll get a corrupted wheel cached. I think perhaps wheels should only be cached + built when coming from a repository of some form and not a direct link (either direct link to a tarball or a file path or what have you).
  • The --cache-dir option isn't controlling where the wheels are being cached too.
  • The --no-cache-dir seems to be somewhat respected, but in a weird way, it tells me it's building Wheels to put into destination None and ultimately the build fails and it installs from sdist.
$ pip install lxml
Collecting lxml
  File was already downloaded /Users/dstufft/Library/Caches/pip/wheels/lxml-3.4.2-cp34-cp34m-macosx_10_10_x86_64.whl
Skipping lxml, due to already being wheel.
Installing collected packages: lxml
Successfully installed lxml-3.4.2
@dstufft

This comment has been minimized.

Show comment
Hide comment
@dstufft

dstufft Apr 7, 2015

Member

Some of the issues above could be solved by fixing #2657 and then using that option.

Member

dstufft commented Apr 7, 2015

Some of the issues above could be solved by fixing #2657 and then using that option.

@glyph

This comment has been minimized.

Show comment
Hide comment
@glyph

glyph Apr 7, 2015

Thanks for submitting this, @rbtcollins. It definitely looks exciting :). Keep in mind, however, that some packages still build corrupt wheels, or wheels with wrong version information – see for example https://bitbucket.org/cffi/cffi/issue/182/building-a-wheel-with-pypy-accidentally – so it needs to be easy to disable this somehow in certain cases.

glyph commented Apr 7, 2015

Thanks for submitting this, @rbtcollins. It definitely looks exciting :). Keep in mind, however, that some packages still build corrupt wheels, or wheels with wrong version information – see for example https://bitbucket.org/cffi/cffi/issue/182/building-a-wheel-with-pypy-accidentally – so it needs to be easy to disable this somehow in certain cases.

@dstufft

This comment has been minimized.

Show comment
Hide comment
@dstufft

dstufft Apr 7, 2015

Member

Looking at the issue @glyph mentioned, at some level we're going to have to tell those people that they need to update their packaging to work in this way. However I don't think we'd want to do that right off the bat.

Perhaps a (temporary) --[no-]implicit-wheel-build flag which defaults to off in 7.0.0 but can be opt in (to give people a chance to try it out without it breaking all the things immediately), then flip the behavior in 8.0.0 so that it defaults to building wheels and users have to opt out. Then in 9.0.0 or so we can deprecate and/or remove those options and make this the only mode of operation.

Another possible option (which could be done in tangent with the first option or instead of) is offer an option which lets people provide the names of projects which should not have implicit wheels built for them.

Member

dstufft commented Apr 7, 2015

Looking at the issue @glyph mentioned, at some level we're going to have to tell those people that they need to update their packaging to work in this way. However I don't think we'd want to do that right off the bat.

Perhaps a (temporary) --[no-]implicit-wheel-build flag which defaults to off in 7.0.0 but can be opt in (to give people a chance to try it out without it breaking all the things immediately), then flip the behavior in 8.0.0 so that it defaults to building wheels and users have to opt out. Then in 9.0.0 or so we can deprecate and/or remove those options and make this the only mode of operation.

Another possible option (which could be done in tangent with the first option or instead of) is offer an option which lets people provide the names of projects which should not have implicit wheels built for them.

@dstufft

This comment has been minimized.

Show comment
Hide comment
@dstufft

dstufft Apr 7, 2015

Member

Another possible issue, currently by dumping Wheels into a directory and adding that to find-links we create a situation where it's basically impossible for an author to ever remove a version from PyPI because anyone who already installed it will have it available inside their wheel cache. This could possibly really confuse people who get two different versions on two different machines and don't really understand why. We could just say that this is something that authors risk when they delete files (and certainly PyPI is currently set up to make deletions a rare event) and leave it at that.

Another option that could also tie into fixes for other things, is to instead iterate over the links of things to install and if we find a non Wheel then we need to attempt to build a wheel and store it in a location that is specific to that exact URL. Then replace the link with a file link to the wheel for that URL. This would mean that instead of relying on find-links we replace a sdist link with a wheel link internally which means that if someone deletes a file from PyPI we won't discover the sdist link anymore and we'll just ignore the wheel file that was generated for it. This could also be used to fix the the problem where different indexes are leaking into each other.

Some psuedo code:

def locate_or_build_wheel(link):
    if os.path.exists(os.path.join(WHEEL_CACHE_DIR, link.url)):
        return Link(..path to wheel file on disk)
    try:
        build_wheel(link)
    except BuildError:
        return link
    else:
        return Link(..path to freshly built wheel file on disk)

links = [locate_or_build_wheel(link) for links]

Does that make sense?

Member

dstufft commented Apr 7, 2015

Another possible issue, currently by dumping Wheels into a directory and adding that to find-links we create a situation where it's basically impossible for an author to ever remove a version from PyPI because anyone who already installed it will have it available inside their wheel cache. This could possibly really confuse people who get two different versions on two different machines and don't really understand why. We could just say that this is something that authors risk when they delete files (and certainly PyPI is currently set up to make deletions a rare event) and leave it at that.

Another option that could also tie into fixes for other things, is to instead iterate over the links of things to install and if we find a non Wheel then we need to attempt to build a wheel and store it in a location that is specific to that exact URL. Then replace the link with a file link to the wheel for that URL. This would mean that instead of relying on find-links we replace a sdist link with a wheel link internally which means that if someone deletes a file from PyPI we won't discover the sdist link anymore and we'll just ignore the wheel file that was generated for it. This could also be used to fix the the problem where different indexes are leaking into each other.

Some psuedo code:

def locate_or_build_wheel(link):
    if os.path.exists(os.path.join(WHEEL_CACHE_DIR, link.url)):
        return Link(..path to wheel file on disk)
    try:
        build_wheel(link)
    except BuildError:
        return link
    else:
        return Link(..path to freshly built wheel file on disk)

links = [locate_or_build_wheel(link) for links]

Does that make sense?

@rbtcollins

This comment has been minimized.

Show comment
Hide comment
@rbtcollins

rbtcollins Apr 7, 2015

Contributor

So, which of these are must-do's and which are 'hmmm it would be nice' ? E.g. the duplicate copy in cache/wheels which fixing #2657 would fix.

@glyph disabling wheel building can be done with --no-cache-dir today - thats deliberate. I can also imagine adding a blacklist feature 'don't build wheels for X'. I'd rather not have to disable it entirely to deal with one bad package.

Per-index wheel caches - that means that we won't cache results against bandersnatch mirrors etc, which is sad. If folk are doing local builds, they should really be annotating them someway (and we should make sure there's a means to do that in the version number). One way we could address the skew issue is to teach PackageFinder about caches - e.g. 'here is a set of links, but only use them when a matching version was found in a different source'. Then we'd put the wheel cache dir in that thing rather than in the regular find-links list.

I'm happy to have a dedicated flag if we think thats needed, but I don't think defaulting it off makes sense: I very much doubt we'll get much in the way of early adopter testing. More effective to offer an easy-opt-out and gather feedback from bug reports.

Re deleting things from pypi - various mirrors already make that ineffective as remediation for bad releases... (Not bandersnatch, but its not the only mirror around). Roll forward my friends, roll forward.

Contributor

rbtcollins commented Apr 7, 2015

So, which of these are must-do's and which are 'hmmm it would be nice' ? E.g. the duplicate copy in cache/wheels which fixing #2657 would fix.

@glyph disabling wheel building can be done with --no-cache-dir today - thats deliberate. I can also imagine adding a blacklist feature 'don't build wheels for X'. I'd rather not have to disable it entirely to deal with one bad package.

Per-index wheel caches - that means that we won't cache results against bandersnatch mirrors etc, which is sad. If folk are doing local builds, they should really be annotating them someway (and we should make sure there's a means to do that in the version number). One way we could address the skew issue is to teach PackageFinder about caches - e.g. 'here is a set of links, but only use them when a matching version was found in a different source'. Then we'd put the wheel cache dir in that thing rather than in the regular find-links list.

I'm happy to have a dedicated flag if we think thats needed, but I don't think defaulting it off makes sense: I very much doubt we'll get much in the way of early adopter testing. More effective to offer an easy-opt-out and gather feedback from bug reports.

Re deleting things from pypi - various mirrors already make that ineffective as remediation for bad releases... (Not bandersnatch, but its not the only mirror around). Roll forward my friends, roll forward.

@rbtcollins

This comment has been minimized.

Show comment
Hide comment
@rbtcollins

rbtcollins Apr 8, 2015

Contributor

The skipping output was due to an inverted if, sorry.

Contributor

rbtcollins commented Apr 8, 2015

The skipping output was due to an inverted if, sorry.

@rbtcollins

This comment has been minimized.

Show comment
Hide comment
@rbtcollins

rbtcollins Apr 8, 2015

Contributor

Your pseudo-code and what I refer to as a cache above are roughly equivalent I think.

Contributor

rbtcollins commented Apr 8, 2015

Your pseudo-code and what I refer to as a cache above are roughly equivalent I think.

@glyph

This comment has been minimized.

Show comment
Hide comment
@glyph

glyph Apr 8, 2015

@rbtcollins:

disabling wheel building can be done with --no-cache-dir today - thats deliberate. I can also imagine adding a blacklist feature 'don't build wheels for X'. I'd rather not have to disable it entirely to deal with one bad package.

--no-cache-dir is unfortunately over-broad, though. Wouldn't that disable all caching? I'd want to still use my download cache even if I can't use my wheel cache.

What I would really prefer is something like pip install some-huge-application --no-wheels-for=cffi,lxml so that wheels of all dependencies but the problematic packages were built automatically.

glyph commented Apr 8, 2015

@rbtcollins:

disabling wheel building can be done with --no-cache-dir today - thats deliberate. I can also imagine adding a blacklist feature 'don't build wheels for X'. I'd rather not have to disable it entirely to deal with one bad package.

--no-cache-dir is unfortunately over-broad, though. Wouldn't that disable all caching? I'd want to still use my download cache even if I can't use my wheel cache.

What I would really prefer is something like pip install some-huge-application --no-wheels-for=cffi,lxml so that wheels of all dependencies but the problematic packages were built automatically.

@rbtcollins

This comment has been minimized.

Show comment
Hide comment
@rbtcollins

rbtcollins Apr 8, 2015

Contributor

@glyph - I'm working on separating out the indices with a lookaside cache rather than merely reusing the find-links code for the cache. If you wanted to push up an additional commit adding a blacklist like you propose, that would be awesome.

Contributor

rbtcollins commented Apr 8, 2015

@glyph - I'm working on separating out the indices with a lookaside cache rather than merely reusing the find-links code for the cache. If you wanted to push up an additional commit adding a blacklist like you propose, that would be awesome.

@glyph

This comment has been minimized.

Show comment
Hide comment
@glyph

glyph Apr 8, 2015

Perhaps at the sprints :).

On Apr 7, 2015, at 22:38, rbtcollins notifications@github.com wrote:

@glyph https://github.com/glyph - I'm working on separating out the indices with a lookaside cache rather than merely reusing the find-links code for the cache. If you wanted to push up an additional commit adding a blacklist like you propose, that would be awesome.


Reply to this email directly or view it on GitHub #2618 (comment).

glyph commented Apr 8, 2015

Perhaps at the sprints :).

On Apr 7, 2015, at 22:38, rbtcollins notifications@github.com wrote:

@glyph https://github.com/glyph - I'm working on separating out the indices with a lookaside cache rather than merely reusing the find-links code for the cache. If you wanted to push up an additional commit adding a blacklist like you propose, that would be awesome.


Reply to this email directly or view it on GitHub #2618 (comment).

@dholth

This comment has been minimized.

Show comment
Hide comment
@dholth

dholth Apr 8, 2015

Member

One of the motivations for wheel was that you would have a cache that you could re-install from which was frustratingly not the case with pip's existing cache of sdists. So it's a feature.

Member

dholth commented Apr 8, 2015

One of the motivations for wheel was that you would have a cache that you could re-install from which was frustratingly not the case with pip's existing cache of sdists. So it's a feature.

@rbtcollins

This comment has been minimized.

Show comment
Hide comment
@rbtcollins

rbtcollins Apr 8, 2015

Contributor

@dholth sorry - what was / is a feature? github has lost the thread I fear :)

Contributor

rbtcollins commented Apr 8, 2015

@dholth sorry - what was / is a feature? github has lost the thread I fear :)

@dholth

This comment has been minimized.

Show comment
Hide comment
@dholth

dholth Apr 8, 2015

Member

It would be a feature that you can install from the cached wheels;
that you can't install from pip's download cache was always a
frustration for me.

On Wed, Apr 8, 2015, at 02:29 PM, rbtcollins wrote:

@dholth[1] sorry - what was / is a feature? github has lost the thread
I fear :)

— Reply to this email directly or view it on GitHub[2].

Links:

  1. https://github.com/dholth
  2. #2618 (comment)
Member

dholth commented Apr 8, 2015

It would be a feature that you can install from the cached wheels;
that you can't install from pip's download cache was always a
frustration for me.

On Wed, Apr 8, 2015, at 02:29 PM, rbtcollins wrote:

@dholth[1] sorry - what was / is a feature? github has lost the thread
I fear :)

— Reply to this email directly or view it on GitHub[2].

Links:

  1. https://github.com/dholth
  2. #2618 (comment)
@dstufft

This comment has been minimized.

Show comment
Hide comment
@dstufft

dstufft Apr 8, 2015

Member

I disagree. A cache that you can install from directly feels like an edge case at best. It's certainly not worth the other problems like leaking versions between indexes.

On Apr 8, 2015, at 1:39 PM, Daniel Holth notifications@github.com wrote:

One of the motivations for wheel was that you would have a cache that you could re-install from which was frustratingly not the case with pip's existing cache of sdists. So it's a feature.


Reply to this email directly or view it on GitHub.

Member

dstufft commented Apr 8, 2015

I disagree. A cache that you can install from directly feels like an edge case at best. It's certainly not worth the other problems like leaking versions between indexes.

On Apr 8, 2015, at 1:39 PM, Daniel Holth notifications@github.com wrote:

One of the motivations for wheel was that you would have a cache that you could re-install from which was frustratingly not the case with pip's existing cache of sdists. So it's a feature.


Reply to this email directly or view it on GitHub.

@rbtcollins

This comment has been minimized.

Show comment
Hide comment
@rbtcollins

rbtcollins Apr 9, 2015

Contributor

Ok:

  • I've fixed the calls to bdist-wheel when wheel isn't installed
  • partially complete (next push will do it) made the wheel cache not trigger the find-links logic, so it won't affect what versions are found.
  • no longer puts existing wheels into the cache (because its now a new code path from 'download' it was easy to put in)
  • honours cache-dir for where to store the wheels

I'm not sure what to do about the direct link vs via a name service. Any VCS repo may have a non-unique version. Perhaps always build an sdist (IIRC there's a PR for that) and key off of a hash of the sdist? That would require downloading the sdist even if there is a cached wheel already present, but would eliminate the failure mode fairly comprehensively I think.

Contributor

rbtcollins commented Apr 9, 2015

Ok:

  • I've fixed the calls to bdist-wheel when wheel isn't installed
  • partially complete (next push will do it) made the wheel cache not trigger the find-links logic, so it won't affect what versions are found.
  • no longer puts existing wheels into the cache (because its now a new code path from 'download' it was easy to put in)
  • honours cache-dir for where to store the wheels

I'm not sure what to do about the direct link vs via a name service. Any VCS repo may have a non-unique version. Perhaps always build an sdist (IIRC there's a PR for that) and key off of a hash of the sdist? That would require downloading the sdist even if there is a cached wheel already present, but would eliminate the failure mode fairly comprehensively I think.

@rbtcollins

This comment has been minimized.

Show comment
Hide comment
@rbtcollins

rbtcollins Apr 9, 2015

Contributor

still pending:

  • test fallout from the stderr thing
  • discussion about handling non-registry based things like direct urls (both vcs and to tarballs etc)
  • blacklist mechanism for controlling lxml etc etc.

I think the first two are needed to merge this, but the third IMO is a nicety - we can add it later: because this can be heavy-handed-disabled via the cache dir, correctness is able to be preserved.

Contributor

rbtcollins commented Apr 9, 2015

still pending:

  • test fallout from the stderr thing
  • discussion about handling non-registry based things like direct urls (both vcs and to tarballs etc)
  • blacklist mechanism for controlling lxml etc etc.

I think the first two are needed to merge this, but the third IMO is a nicety - we can add it later: because this can be heavy-handed-disabled via the cache dir, correctness is able to be preserved.

Show outdated Hide outdated pip/req/req_install.py Outdated
Show outdated Hide outdated pip/wheel.py Outdated
Show outdated Hide outdated pip/wheel.py Outdated
@rbtcollins

This comment has been minimized.

Show comment
Hide comment
@rbtcollins

rbtcollins Apr 10, 2015

Contributor

This now has a limit on it to not build or cache wheels for unversioned things. E.g. unless the url looks like an archive, we won't cache for it at all, which means that e.g. git+https://github.com/pypa/pip will not get cached wheels built. But a url like https://github.com/pypa/project-2 will. This isn't perfect, but it should be enough to avoid the vast majority of potential breakage.

Contributor

rbtcollins commented Apr 10, 2015

This now has a limit on it to not build or cache wheels for unversioned things. E.g. unless the url looks like an archive, we won't cache for it at all, which means that e.g. git+https://github.com/pypa/pip will not get cached wheels built. But a url like https://github.com/pypa/project-2 will. This isn't perfect, but it should be enough to avoid the vast majority of potential breakage.

@rbtcollins

This comment has been minimized.

Show comment
Hide comment
@rbtcollins

rbtcollins Apr 13, 2015

Contributor
  self.do_handshake()
File "/opt/python/3.2.5/lib/python3.2/ssl.py", line 451, in do_handshake
  self._sslobj.do_handshake()

ssl.SSLError: [Errno 8] _ssl.c:392: EOF occurred in violation of protocol

  • network failures.
Contributor

rbtcollins commented Apr 13, 2015

  self.do_handshake()
File "/opt/python/3.2.5/lib/python3.2/ssl.py", line 451, in do_handshake
  self._sslobj.do_handshake()

ssl.SSLError: [Errno 8] _ssl.c:392: EOF occurred in violation of protocol

  • network failures.

Robert Collins added some commits Mar 30, 2015

Robert Collins
Issue #2563: Read cached wheels from ~/.cache/pip
This won't put wheels into that directory, but will read them if they
are there. --no-cache-dir will disable reading such wheels.
Robert Collins
Issue #2140: Build wheels automatically
Building wheels before installing elminates a cause of broken environments -
where install fails after we've already installed one or more packages.

If a package fails to wheel, we run setup.py install as normally.
Robert Collins
Break out version extraction for reuse.
This is needed for determining the wheel-cachability of a requirement.
requirement_set,
finder,
build_options=[],
global_options=[],

This comment has been minimized.

@xavfernandez

xavfernandez Apr 13, 2015

Contributor

Wondering if this is not an issue (for build_options and global_options)... maybe the use of these would disable the wheel caching as I dont think we want to build wheels for all possible options ?

@xavfernandez

xavfernandez Apr 13, 2015

Contributor

Wondering if this is not an issue (for build_options and global_options)... maybe the use of these would disable the wheel caching as I dont think we want to build wheels for all possible options ?

@dstufft

This comment has been minimized.

Show comment
Hide comment
@dstufft

dstufft Apr 13, 2015

Member

I'm going to go ahead and merge this. Some things may need to be adjusted still but I don't think that we're going to fully hammer them out with this sitting in a branch, and having it sit in a branch just makes it harder on @rbtcollins to have to keep updating it. I do think that the overall structure is sound and that the general feature is something we want.

Member

dstufft commented Apr 13, 2015

I'm going to go ahead and merge this. Some things may need to be adjusted still but I don't think that we're going to fully hammer them out with this sitting in a branch, and having it sit in a branch just makes it harder on @rbtcollins to have to keep updating it. I do think that the overall structure is sound and that the general feature is something we want.

if not wheel.supported():
# Built for a different python/arch/etc
continue
candidates.append((wheel.support_index_min(), wheel_name))

This comment has been minimized.

@qwcode

qwcode Apr 13, 2015

Contributor

I was specifically looking for this sorting by the support index. yay, it's here.

@qwcode

qwcode Apr 13, 2015

Contributor

I was specifically looking for this sorting by the support index. yay, it's here.

dstufft added a commit that referenced this pull request Apr 13, 2015

Merge pull request #2618 from rbtcollins/issue-2140
Issue #2140: Build wheels during pip install

@dstufft dstufft merged commit d043e4b into pypa:develop Apr 13, 2015

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
@qwcode

This comment has been minimized.

Show comment
Hide comment
@qwcode

qwcode Apr 13, 2015

Contributor

to be clear, I don't think we could release without the blacklist, so I assume that's coming.

Contributor

qwcode commented Apr 13, 2015

to be clear, I don't think we could release without the blacklist, so I assume that's coming.

@dstufft

This comment has been minimized.

Show comment
Hide comment
@dstufft

dstufft Apr 13, 2015

Member

Added #2675 to track that.

Member

dstufft commented Apr 13, 2015

Added #2675 to track that.

@xavfernandez

This comment has been minimized.

Show comment
Hide comment
@xavfernandez

xavfernandez Apr 13, 2015

Contributor

The *_options issue seems important also:

pip install foo --install-option some_option

will end up building the wheel and installing from it without the passed option...

Contributor

xavfernandez commented Apr 13, 2015

The *_options issue seems important also:

pip install foo --install-option some_option

will end up building the wheel and installing from it without the passed option...

@dstufft

This comment has been minimized.

Show comment
Hide comment
@dstufft

dstufft Apr 13, 2015

Member

So, that's actually already the case if you're installing from Wheels, they just get flat out ignored. We should probably figure something out to deal with them though.

Member

dstufft commented Apr 13, 2015

So, that's actually already the case if you're installing from Wheels, they just get flat out ignored. We should probably figure something out to deal with them though.

@dstufft

This comment has been minimized.

Show comment
Hide comment
@dstufft

dstufft Apr 13, 2015

Member

Also added #2676 and #2677

Member

dstufft commented Apr 13, 2015

Also added #2676 and #2677

@rbtcollins rbtcollins deleted the rbtcollins:issue-2140 branch Apr 22, 2015

@nils-werner

This comment has been minimized.

Show comment
Hide comment
@nils-werner

nils-werner Jun 3, 2015

Is there a way to make pip create wheels and store them in the cache dir without actually installing them? I would like to fill my wheel cache from time to time too...

nils-werner commented Jun 3, 2015

Is there a way to make pip create wheels and store them in the cache dir without actually installing them? I would like to fill my wheel cache from time to time too...

@akaihola

This comment has been minimized.

Show comment
Hide comment
@akaihola

akaihola Nov 25, 2015

Contributor

Parameter name unpack in the docstring doesn't match autobuilding in the function signature.

Contributor

akaihola commented on pip/wheel.py in 08acb66 Nov 25, 2015

Parameter name unpack in the docstring doesn't match autobuilding in the function signature.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment