Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Issue #2140: Build wheels during pip install #2618

Merged
merged 3 commits into from Apr 13, 2015
Merged

Conversation

@rbtcollins
Copy link
Contributor

@rbtcollins rbtcollins commented Mar 30, 2015

This branch is building up to tackle issue #2140 - build wheels during install and then install those wheels.

@rbtcollins
Copy link
Contributor Author

@rbtcollins rbtcollins commented Mar 30, 2015

The topo sort thuings are from pull #2616

@rbtcollins rbtcollins force-pushed the rbtcollins:issue-2140 branch from d73c84f to aec03e0 Mar 30, 2015
@sigmavirus24
Copy link
Member

@sigmavirus24 sigmavirus24 commented Mar 30, 2015

This sounds exciting

@rbtcollins rbtcollins force-pushed the rbtcollins:issue-2140 branch 4 times, most recently from 2e35a5c to 4c89e07 Mar 30, 2015
@alex
Copy link
Member

@alex alex commented Mar 31, 2015

Can all the ensure_dir changes by refactored out into their own PR?

@rbtcollins
Copy link
Contributor Author

@rbtcollins rbtcollins commented Mar 31, 2015

They're a single commit. Can pull it out if desired.

@rbtcollins rbtcollins force-pushed the rbtcollins:issue-2140 branch 4 times, most recently from 758b819 to 82aba41 Mar 31, 2015
@rbtcollins
Copy link
Contributor Author

@rbtcollins rbtcollins commented Apr 1, 2015

Its mainly tested via all the existing tests, but I added an explicit test looking at the output to be sure building was happening and handling of non-wheelable things.

@rbtcollins rbtcollins force-pushed the rbtcollins:issue-2140 branch 11 times, most recently from c2c03a1 to affd2bf Apr 1, 2015
@rbtcollins
Copy link
Contributor Author

@rbtcollins rbtcollins commented Apr 3, 2015

I'm not sure why we're seeing those failures. Its not as simple as hashseed - I've tried using the same hashseed locally. It might be an isolation failure, or something. There's obviously an underlying issue, but until I can provoke it I can't fix it :(

I'm going to try adding diagnostic changes on top of the stack to do that.

@rbtcollins rbtcollins force-pushed the rbtcollins:issue-2140 branch 2 times, most recently from c08a7f0 to e93b9ed Apr 3, 2015
try:
os.makedirs(path)
except OSError as e:
if e.errno == errno.EEXIST:

This comment has been minimized.

@berkerpeksag

berkerpeksag Apr 3, 2015
Member

or a shorter version:

if e.errno != errno.EEXIST:
    raise
@rbtcollins rbtcollins force-pushed the rbtcollins:issue-2140 branch from e93b9ed to b3229d8 Apr 6, 2015
@rbtcollins
Copy link
Contributor Author

@rbtcollins rbtcollins commented Apr 10, 2015

This now has a limit on it to not build or cache wheels for unversioned things. E.g. unless the url looks like an archive, we won't cache for it at all, which means that e.g. git+https://github.com/pypa/pip will not get cached wheels built. But a url like https://github.com/pypa/project-2 will. This isn't perfect, but it should be enough to avoid the vast majority of potential breakage.

@rbtcollins rbtcollins force-pushed the rbtcollins:issue-2140 branch 3 times, most recently from 869a222 to 4a25059 Apr 12, 2015
@rbtcollins
Copy link
Contributor Author

@rbtcollins rbtcollins commented Apr 13, 2015

  self.do_handshake()
File "/opt/python/3.2.5/lib/python3.2/ssl.py", line 451, in do_handshake
  self._sslobj.do_handshake()

ssl.SSLError: [Errno 8] _ssl.c:392: EOF occurred in violation of protocol

  • network failures.
Robert Collins added 3 commits Apr 10, 2015
This is needed for determining the wheel-cachability of a requirement.
This won't put wheels into that directory, but will read them if they
are there. --no-cache-dir will disable reading such wheels.
Robert Collins
Building wheels before installing elminates a cause of broken environments -
where install fails after we've already installed one or more packages.

If a package fails to wheel, we run setup.py install as normally.
@rbtcollins rbtcollins force-pushed the rbtcollins:issue-2140 branch from 4a25059 to 08acb66 Apr 13, 2015
requirement_set,
finder,
build_options=[],
global_options=[],

This comment has been minimized.

@xavfernandez

xavfernandez Apr 13, 2015
Member

Wondering if this is not an issue (for build_options and global_options)... maybe the use of these would disable the wheel caching as I dont think we want to build wheels for all possible options ?

@dstufft
Copy link
Member

@dstufft dstufft commented Apr 13, 2015

I'm going to go ahead and merge this. Some things may need to be adjusted still but I don't think that we're going to fully hammer them out with this sitting in a branch, and having it sit in a branch just makes it harder on @rbtcollins to have to keep updating it. I do think that the overall structure is sound and that the general feature is something we want.

if not wheel.supported():
# Built for a different python/arch/etc
continue
candidates.append((wheel.support_index_min(), wheel_name))

This comment has been minimized.

@qwcode

qwcode Apr 13, 2015
Contributor

I was specifically looking for this sorting by the support index. yay, it's here.

dstufft added a commit that referenced this pull request Apr 13, 2015
Issue #2140: Build wheels during pip install
@dstufft dstufft merged commit d043e4b into pypa:develop Apr 13, 2015
1 check passed
1 check passed
continuous-integration/travis-ci/pr The Travis CI build passed
Details
@qwcode
Copy link
Contributor

@qwcode qwcode commented Apr 13, 2015

to be clear, I don't think we could release without the blacklist, so I assume that's coming.

@dstufft
Copy link
Member

@dstufft dstufft commented Apr 13, 2015

Added #2675 to track that.

@xavfernandez
Copy link
Member

@xavfernandez xavfernandez commented Apr 13, 2015

The *_options issue seems important also:

pip install foo --install-option some_option

will end up building the wheel and installing from it without the passed option...

@dstufft
Copy link
Member

@dstufft dstufft commented Apr 13, 2015

So, that's actually already the case if you're installing from Wheels, they just get flat out ignored. We should probably figure something out to deal with them though.

@dstufft
Copy link
Member

@dstufft dstufft commented Apr 13, 2015

Also added #2676 and #2677

@rbtcollins rbtcollins deleted the rbtcollins:issue-2140 branch Apr 22, 2015
@nils-werner
Copy link

@nils-werner nils-werner commented Jun 3, 2015

Is there a way to make pip create wheels and store them in the cache dir without actually installing them? I would like to fill my wheel cache from time to time too...

@akaihola

This comment has been minimized.

Copy link
Contributor

@akaihola akaihola commented on pip/wheel.py in 08acb66 Nov 25, 2015

Parameter name unpack in the docstring doesn't match autobuilding in the function signature.

@lock lock bot added the S: auto-locked label Jun 4, 2019
@lock lock bot locked as resolved and limited conversation to collaborators Jun 4, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

You can’t perform that action at this time.