-
Notifications
You must be signed in to change notification settings - Fork 237
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
revert PR9; add --no-deps to pip wheel; address PR11 #18
Conversation
@YannickJadoul would you test this? |
@tgarc Yes, please! Thanks. I'll let you know. If you want, I'd be curious to have your input on the solution to the other issues in PR #17, too. EDIT: If you want to follow live, here's my build on Travis CI :-) |
Seems to be working! :-) 2 questions, if I may:
|
@YannickJadoul thanks for testing!
Unfortunately I caused some confusion with my previous PR. Dependencies aren't usually required to do a wheel build, only to install that wheel. (Although some builds do require 'build-dependencies'). In my case doing a pre- pip install was fixing an issue by side-effect that was actually being caused by including setup_requires=['cffi']. (This seems like a bug in cffi or setuptools etc). IOW, my PR was only ever a workaround for a cffi issue.
Okay so now that no wheels are being built for dependencies I had to remove the |
8e22fef
to
7a2e6a2
Compare
Okay, think I've cleaned up as much as I can. @joerick take a look when you have time. |
Ok, perfect :-) That makes things easier than I expected than, indeed :-)
Yeps, that does. Weirdly enough though, I had tried that myself, before, and had even been looking up in the pip source (https://github.com/pypa/pip/blame/21be153044a7aa245e12ce3f86793e9b17201519/pip/index.py#L464-L468) and thought that local files díd get precedence. But I was just curious. This is not a monster hack; it's pretty neat, actually. Just a pity you nééded a workaround. |
Aaaaaaaaaah, I've actually just been testing (with In this case, there is already a |
cibuildwheel/windows.py
Outdated
@@ -17,11 +17,11 @@ def build(project_dir, package_name, output_dir, test_command, test_requires, be | |||
request = urlopen('https://github.com/ogrisel/python-appveyor-demo/raw/09a1c8672e5015a74d8f69d07add6ee803c176ec/appveyor/run_with_env.cmd') | |||
f.write(request.read()) | |||
|
|||
def shell(args, env=None, cwd=None): | |||
def shell(args, env=None, cwd=None, **kwargs): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
kwargs
does not seem to be used anymore?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good catch! Thanks.
cibuildwheel/macos.py
Outdated
|
||
temp_wheel = glob(temp_wheel_dir+'/*.whl')[0] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is there a chance that multiple wheels get built?
I don't see how or why one package could do that, but the Linux version is looping over all wheels. So for consistency we can maybe either change this line, or the one in linux.py?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is still an open question. I still don't know of any projects where multiple wheels get built so I decided not to make any changes, at least for this PR.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok, fair enough. Fine with me.
I have just realised that "for whl in /tmp/linux_wheels/*.whl; do" is probably just easier to write the pattern match, and not per se to do that over all wheels.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've not seen any packages that make more than one wheel, so I think this is a good assumption 👍
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks tgarc! I think maybe removing the find_wheels bits and going back the previous install command would simplify this a bit?
cibuildwheel/util.py
Outdated
sys.stdout.write(\ | ||
' '.join(f for f in itertools.chain(*map(glob.iglob, sys.argv[1:])) \ | ||
if pip.wheel.Wheel(f).supported()))\ | ||
''' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm finding this quite hard to understand. What does this do, and might there be a more elegant way to do it? I'm not averse to separate code for each platform by the way!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also I think this is an undocumented Pip API... Might change in the future.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry, lack of commenting. I can add some.
This is a short python program that takes as arguments a list of wheels and prints out only those wheels that are supported on the executor's platform. I put it here since all platforms use it and there's no need to have it differ per-platform.
But if you're asking why...hopefully my previous comment makes it clear.
Also I think this is an undocumented Pip API... Might change in the future.
I will try and verify whether that's the case. I tend to assume anything not starting with an underscore is fairly stable but this may not be true in pip world.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry, I had missed that previous comment! Yes I think you're right in that respect.
cibuildwheel/linux.py
Outdated
@@ -77,8 +74,7 @@ def build(project_dir, package_name, output_dir, test_command, test_requires, be | |||
# Install packages and test | |||
for PYBIN in {pybin_paths}; do | |||
# Install the wheel we just built | |||
"$PYBIN/pip" install {package_name} \ | |||
--upgrade --force-reinstall --no-deps --no-index -f /output | |||
"$PYBIN/pip" install $("$PYBIN/python" -c "{filter_wheels}" /tmp/linux_wheels/*.whl) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What's the reason for this change? I think the previous one is clearer and also it installs the delocated wheel, which is probably the right thing to do!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yikes! you're absolutely right, I meant to point to the delocated wheel. See my previous comment for why I made this change though.
EDIT fixed; correctly points to delocated wheel now
That's accurate. In fact it says this in the pip docs:
|
I got a chance to look at this again - I'm wondering if a simpler solution to the filter_wheels business would be to delocate/audit into a directory made by I'm sorry to be slowing things down! I just think that it's pretty crucial for the project to try and keep this code simple so that people can peek under the hood when things go wrong and see how it works :) |
No worries, I'm busy too. Thanks for sharing your code. So use a temp directory for installing...interesting. I think I can do that. I still think that the filter_wheels functionality may come in handy at some point but if we can do it your way...sounds cleaner. EDIT Ah, I forgot linux does things differently - it builds all the wheels before installing any of them. I'll have to add some bash code to make this work then. |
@joerick here's a shot at doing what you suggested. Note one significant change: the linux build now more closely follows builds on macos/windows; it uses a single for loop to build/delocate/install/test each wheel. One thing about this that irks me a bit is that if ever Anyway, the tests are failing and I have only tested with windows so I'll need to fix this up. |
bah. this has turned out to be more work than anticipated. seems like for some reason the last step (shutil.move(temp_wheel, output)) is failing on windows & macos. Feel free to chime in if you have thoughts. @joerick is this what you envisioned? Does it seem simpler to you ? |
@tgarc Just to debug, could you maybe |
883802d
to
6e76dc8
Compare
@YannickJadoul found the bug...I needed to add a step to create the output directory. Previously the output directory was getting automatically created by one of the pip commands |
@joerick okay I've found the bugs that I had. The TL;DR is that I had to create extra temp directories for delocated wheels, and manually delete any temporary wheels for each build so they don't get picked up on subsequent builds. The complexity I think is about the same as with filter_wheels but I think the code 'looks' simpler. Would you give it a look over for sanity? Then I'll squash the commits and merge it. |
@tgarc Nice! My test build is also succeeding: https://travis-ci.org/YannickJadoul/Parselmouth/jobs/251766087 |
Sweet! I'm taking a look now. |
Hey @tgarc - this looks good, I just did a few naming tweaks and changes to the temp dir stuff, hopefully that makes the code clearer. Could you pull from my |
741d0cd
to
638d854
Compare
thanks @joerick |
revert PR9; add --no-deps to pip wheel Revert PR9; it was never actually needed in the first place it just fixed an issue by side effect. 1) add --no-deps to pip wheel step in order to avoid building wheels for dependencies. This also addresses the issues raised in PR pypa#11. 2) Modify the pip install step across platforms to allow installation of dependencies from the index. (This is required since we will no longer be building wheels for dependencies).
🎉 |
just released 0.4.0 with these changes. 👍 |
This is a WIP - there will be some tweaking but the proof of concept is there and should be fully functional. I've tested this with my package on all platforms. This should address the issue raised by PR #11 without needed an additional pip pre-install step. It also removes the step of building wheels for dependencies - a step which adds extra build time and may even cause issues for some packages. BTW this follows more closely the build steps in the https://github.com/matthew-brett/multibuild project.
Revert PR9; it was never actually needed in the first place it just fixed an
issue by side effect.
add --no-deps to pip wheel step in order to avoid building wheels for
dependencies. This also addresses the issues raised in PR Solving problems in OS X when library has dependencies #11.
Modify the pip install step across platforms to allow installation of
dependencies from the index. (This is required since we will no longer be
building wheels for dependencies).