Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Any interest in automated builds for OSX wheels? #766

Closed
matthew-brett opened this issue Jul 3, 2014 · 56 comments

Comments

Projects
None yet
4 participants
@matthew-brett
Copy link
Contributor

commented Jul 3, 2014

I posted this to the image-sig mailing list but can't see any evidence it got there. Sorry if this is a duplicate.

I spent some time making a travis-ci repository that will create an
OSX wheel from a Pillow release (the current 2.4.0 release):

https://github.com/matthew-brett/pillow-wheels

When I push commits or trigger a build on this repo, for Python 2.7,
3.3, 3.4 it:

  • builds the dependent libraries
  • builds Pillow
  • copies the needed external libraries into the Pillow distribution
  • makes an OSX wheel
  • installs the wheel into a Python virtualenv
  • runs the unit tests
  • uploads the wheel to a rackspace container

http://a365fff413fe338398b6-1c8a9b3114517dc5fe17b7c3f8c63a43.r19.cf2.rackcdn.com/

Example build etc log here:

https://travis-ci.org/matthew-brett/pillow-wheels

In particular, here's the tests passing on one of the three builds:

https://travis-ci.org/matthew-brett/pillow-wheels/jobs/28768481#L6645

Would y'all be interested to take this over at the python-Pillow
organization? I can set you up with the rackspace upload credentials
for the repo if you are interested.

@aclark4life

This comment has been minimized.

Copy link
Member

commented Jul 3, 2014

@matthew-brett Cool! I don't know about "take this over" but I'd certainly be interested in discussing how we can incorporate this technology… in the mean time if you want to build OS X wheels for 2.5.0 I'll upload them to PyPI

@matthew-brett

This comment has been minimized.

Copy link
Contributor Author

commented Jul 3, 2014

Sure - new wheels here : http://wheels.scikit-image.org/

@matthew-brett

This comment has been minimized.

Copy link
Contributor Author

commented Jul 3, 2014

I'm happy to build wheels from my repo if you like. The alternative is for you to take the wheel-building repo into the python-Pillow organization so the organization maintainers can trigger wheel building as part of the release process. That's what happened with scikit-image...

@wiredfool

This comment has been minimized.

Copy link
Member

commented Jul 3, 2014

A couple of things:

  • Sweet!
  • Since you're using Travis, it looks like we could do continuous integration as well.
  • We have a organizational RackSpace account so we can use that. (we need to add that to the website)
  • This is very nearly what to do on windows as well.
@matthew-brett

This comment has been minimized.

Copy link
Contributor Author

commented Jul 3, 2014

  • It is sweet - good ol' travis
  • You mean add OSX continuous testing for the main repo? Yes, but I think you have to ask travis-ci to turn on multi-os support and adapt your travis-ci script - see http://blog.travis-ci.com/2014-05-13-multi-os-feature-available/
  • Do you want to take over the travis wheels repo then? In which case someone with your rackspace API key would need to run the API key encryption into the .travis.yml file using (say) travis setup cloudfiles : http://docs.travis-ci.com/user/deployment/cloudfiles/
  • What do you do on Windows? It looks as though you've aced your Windows installers.
@aclark4life

This comment has been minimized.

Copy link
Member

commented Jul 3, 2014

On Windows we rely on the kind services of @cgohlke

@wiredfool

This comment has been minimized.

Copy link
Member

commented Jul 3, 2014

Sorry about the missing words, I've been trying out speech recognition and didn't edit well. rsi sucks.

Looking at the blog post, multi-os might not be the right idea right now, since we've got a pretty big build matrix.

Currently windows is @cgohlke doing the builds. He does a great job, but the busfactor there is 1. Automated builds (and ci) is something that I've been working on there, but haven't actually gotten completely working yet.

(and by completely working, I mean, I've got one build sorta working, msvc10-x64, sans a couple of libraries, and the rest fail)

@aclark4life

This comment has been minimized.

Copy link
Member

commented Jul 3, 2014

Thanks for the wheels! Any chance you could do Python 2.6 too for consistency?
screenshot 2014-07-03 14 00 53

@aclark4life

This comment has been minimized.

Copy link
Member

commented Jul 3, 2014

👯 CI all the things

@wiredfool

This comment has been minimized.

Copy link
Member

commented Jul 3, 2014

Yes. CI good. I'd like to see at least periodic build and test of all the supported platforms. Nightlies would work, each commit to master would rock.

Diving deeper into this, it's complicated enough that there's no way that I'd want to do hack this in to our current travis setup. So nightlies might be what we can pull off for now.

I've sent a PR for running the full test suite, in addition to just the selftest.

There's also a patch for pillow for homebrew/py3. Is that something that should be upstreamed?

@wiredfool

This comment has been minimized.

Copy link
Member

commented Jul 3, 2014

I'm +1 on pulling this into the python-pillow org and putting it on our RS key/account. Any objections?

@hugovk

This comment has been minimized.

Copy link
Member

commented Jul 3, 2014

I'm +1 on pulling this into the python-pillow org and putting it on our RS key/account. Any objections?

+1 It'd be good to give @matthew-brett access to it too.

@hugovk

This comment has been minimized.

Copy link
Member

commented Jul 3, 2014

Yes. CI good. I'd like to see at least periodic build and test of all the supported platforms. Nightlies would work, each commit to master would rock.

Diving deeper into this, it's complicated enough that there's no way that I'd want to do hack this in to our current travis setup. So nightlies might be what we can pull off for now.

I wonder, is it possible to kick off a completely different Travis build from the main repo? Or similarly, nudge a change in another repo so kick off another Travis build?

A quick search brings up this:

Q: How can I set up a build schedule for Travis-CI that is not based around pushing to GitHub? I want to run Selenium tests against my production site nightly. I'm happy with a hacky solution if official support is not present.

A: This feature has been requested some time ago, but is not yet being worked on. travis-ci/travis-ci#582

Here is an external solution that will trigger new builds. https://traviscron.pythonanywhere.com/

Alternatively, a simple nightly cron job to run git commit --allow-empty then git push will nudge a Travis run without changing anything (idea via).

@wiredfool

This comment has been minimized.

Copy link
Member

commented Jul 3, 2014

Ok, Forked in to python-pillow. I think I've got permissions setup so that @matthew-brett and @hugovk can push to it, please check and let me know if I've messed something up there.

I'll get the key setup so that we're pushing builds to the pillow RS account.

@wiredfool

This comment has been minimized.

Copy link
Member

commented Jul 3, 2014

travis_ping from here: https://github.com/FiloSottile/travis-cron, which is part of traviscron, looks like just the ticket for nightly builds. I can hang it off of a machine somwhere easily enough.

@hugovk

This comment has been minimized.

Copy link
Member

commented Jul 3, 2014

Yep, looks like I can edit ok.

I'm not sure if the distinction is important for us, but rather than forking it's possible to transfer a repo like this:
https://help.github.com/articles/how-to-transfer-a-repository

@wiredfool

This comment has been minimized.

Copy link
Member

commented Jul 3, 2014

Ok, I guess I didn't get the distinction. Not sure that it matters though.

@hugovk

This comment has been minimized.

Copy link
Member

commented Jul 3, 2014

Poking around, just ran up against this:

Sorry, forked repositories are not currently searchable.

You could try searching the parent repository.

https://github.com/python-pillow/pillow-wheels/search?q=get_python_environment&type=Code

@aclark4life

This comment has been minimized.

Copy link
Member

commented Jul 3, 2014

Yeah I'd prefer a transfer I think

@wiredfool

This comment has been minimized.

Copy link
Member

commented Jul 3, 2014

ok. I'll kill the fork and we can do the transfer.

@wiredfool

This comment has been minimized.

Copy link
Member

commented Jul 3, 2014

it is now an ex-fork.

@hugovk

This comment has been minimized.

Copy link
Member

commented Jul 3, 2014

Instructions for transferring from a user to an org:

Transferring from a user to an organization

Users must have admin or owner rights within the receiving organization before they can transfer a repository that they individually own. If the user does not already have this level of access, a temporary admin team can be created with only the user. The user sending the repository is the only one who can perform the transfer.

Just as when transferring a repository between two organizations, when a repository is transferred from a user to an organization, the Owners team is the only group with read/write access to the repository, but they can choose to add other teams for further access. Our article on permission levels in organizations has more information.

https://help.github.com/articles/how-to-transfer-a-repository#transferring-from-a-user-to-an-organization

@matthew-brett

This comment has been minimized.

Copy link
Contributor Author

commented Jul 4, 2014

No problem to transfer - I will get the tests running OK first though, with your pull-requests.

Someone who knows your rackspace credentials should then update the travis encrypted key in the .travis.yml file...

@wiredfool

This comment has been minimized.

Copy link
Member

commented Jul 4, 2014

I'll do the key setup, but likely it's going to be tomorrow. Got to take kiddos to fireworks tonight.

@matthew-brett

This comment has been minimized.

Copy link
Contributor Author

commented Jul 5, 2014

I made a mistake building the first set of wheels; I believe they will not work correctly on OSX 10.6. I have uploaded new ones to http://wheels.scikit-image.org/ (via an automated build). Would someone mind replacing the pypi wheels with these ones?

@aclark4life

This comment has been minimized.

Copy link
Member

commented Jul 5, 2014

That would be me, done! Thanks

@matthew-brett

This comment has been minimized.

Copy link
Contributor Author

commented Jul 5, 2014

Thanks for that.

The extended tests are locking up or timing out on the travis machines, with various parameters, such as:

NOSE_PROCESSES=0 (https://travis-ci.org/matthew-brett/pillow-wheels/builds/29108440)

or

NOSE_PROCESS_TIMEOUT=600 (https://travis-ci.org/matthew-brett/pillow-wheels/builds/29140070)

Same tests pass on my laptop though. I'm not sure what else to check - any suggestions?

@wiredfool

This comment has been minimized.

Copy link
Member

commented Jul 5, 2014

@matthew-brett Is it just the JPEG2000 tests that are failing? It looks like that's a developing, if not well known, issue. If so, give this branch a shot: https://github.com/wiredfool/Pillow/tree/osx-skip-jpeg2k using the brand new skip bad test functionality.

@matthew-brett

This comment has been minimized.

Copy link
Contributor Author

commented Jul 6, 2014

@matthew-brett

This comment has been minimized.

Copy link
Contributor Author

commented Jul 6, 2014

I'm happy to transfer this guy to you whenever you are ready.

@wiredfool

This comment has been minimized.

Copy link
Member

commented Jul 6, 2014

@matthew-brett: I've set the pillow-wheel team to admin level. That should be what you need to be able to transfer the repo.

@matthew-brett

This comment has been minimized.

Copy link
Contributor Author

commented Jul 6, 2014

Thanks. I've transferred the repo; can someone now activate testing on travis-ci?

@wiredfool

This comment has been minimized.

Copy link
Member

commented Jul 7, 2014

Not quite getting the secure key working correctly.

@aclark4life

This comment has been minimized.

Copy link
Member

commented Jul 7, 2014

Travis looks enabled to me
screenshot 2014-07-07 12 06 50

@matthew-brett

This comment has been minimized.

Copy link
Contributor Author

commented Jul 24, 2014

Guys - can I help with the auth ? Is it possible you have the wrong rackspace region (not ORD)? If you like I can put a pull request with the scikit-learn rackspace account stuff.

@wiredfool

This comment has been minimized.

Copy link
Member

commented Jul 24, 2014

I've got a container named "wheels" in ORD. I've uploaded a test file: https://c3dc8d2a82eed617b4ac-6a3b6a0f5f6ab91aadc447b2a897dd9a.ssl.cf2.rackcdn.com/index.html

I've got a user named travis-upload, which has access to files to view/create/edit/delete.

I've taken the api key, and run:

travis encrypt --add deploy.api-key

And got the secure: line in .travis.yml.

I think I've hit the obvious issues.

@wiredfool

This comment has been minimized.

Copy link
Member

commented Jul 24, 2014

Ah, I think I was actually on my fork, not one where origin was the python-pillow version.

@wiredfool

This comment has been minimized.

Copy link
Member

commented Jul 24, 2014

Looks like that was it, files were built and uploaded.

@matthew-brett

This comment has been minimized.

Copy link
Contributor Author

commented Jul 25, 2014

I see that the current release does not have the wheels. Can I help?

@aclark4life

This comment has been minimized.

Copy link
Member

commented Jul 25, 2014

Please! Thanks

@matthew-brett

This comment has been minimized.

Copy link
Contributor Author

commented Jul 25, 2014

I was hoping y'all would have the files you needed in your rackspace container. Is that true? Then, is there anything I can do to make it easier to upload those to pypi? Otherwise I'll build them myself now..

@aclark4life

This comment has been minimized.

Copy link
Member

commented Jul 25, 2014

I haven't looked into this yet but if you want to point me at the wheels I'll upload them to PyPI

@matthew-brett

This comment has been minimized.

Copy link
Contributor Author

commented Jul 25, 2014

I think your pillow-wheels repo has already built files and put them into your rackspace container, but I can't log into your rackspace account to check. The public link above is https://c3dc8d2a82eed617b4ac-6a3b6a0f5f6ab91aadc447b2a897dd9a.ssl.cf2.rackcdn.com/index.html - but even if you delete the index.html you can't see the directory listing unless you are logged into the account. It's a presidential PITA to enable the public directory listing on a rackspace container - I can help you with that if you would like, but the easiest would be if you logged into the rackspace account and checked the directory to see if the wheels are already there.

@aclark4life

This comment has been minimized.

Copy link
Member

commented Jul 25, 2014

I only see 2.5.0
screenshot 2014-07-24 22 33 16

@matthew-brett

This comment has been minimized.

Copy link
Contributor Author

commented Jul 25, 2014

Oops - that's my fault : python-pillow/pillow-wheels#5

I'd personally suggest that you

  • delete index.html for the wheels container (with the web interface)
  • set the TTL (time to live) to the minimum value for the container (with the web interface)

Then turn on public directory listings for the container - if you can. Something like this might work:

from __future__ import print_function

from os.path import expanduser, join as pjoin

import pyrax

pyrax.set_setting('identity_type', 'rackspace')
home = expanduser('~')
pyrax.set_credential_file(pjoin(home, ".scikit_creds"), region="ORD")
""" Creds file looks like this
[rackspace_cloud]
username = your-username
api_key = abcd1234abcd1234etc
"""
cf = pyrax.connect_to_cloudfiles(region="ORD")
wheels_container = cf.get_container("wheels")
print(wheels_container.get_metadata())
wheels_container.set_metadata({'x-container-meta-web-listings': 'true'})
print(wheels_container.get_metadata())

(as a way of automating the horror here: https://community.rackspace.com/products/f/25/t/1736)

If that works then y'all (and we all) will be able see the generated files soon after they are built, and it's easy to upload them to pypi with curl and twine without logging into the rackspace account.

@wiredfool

This comment has been minimized.

Copy link
Member

commented Jul 25, 2014

Because that's intuitiative. I was poking at the web interface today, trying to get it to work.

@matthew-brett

This comment has been minimized.

Copy link
Contributor Author

commented Jul 25, 2014

The rackspace interface for enabling directory listing is extraordinary ... Let me know if the pyrax code works for you.

The travis build queue went down for an hour or two, now coming back up.

@wiredfool

This comment has been minimized.

Copy link
Member

commented Jul 25, 2014

I'm getting a sandbox violation installing pyrax -- from python-novaclient within a virtualenv where I've got pillow, works in system python.

 python set-listable.py 
{'web_error': 'error.html', 'access_control_expose_headers': 'etag location x-timestamp x-trans-id', 'web_index': 'index.html', 'access_log_delivery': 'false'}
{'web_error': 'error.html', 'access_control_expose_headers': 'etag location x-timestamp x-trans-id', 'web_listings': 'true', 'access_log_delivery': 'false', 'web_index': 'index.html'}

I think that's right, we'll know in 10 minutes or so.

aaaandddd. no. Blank. Maybe the old timeout needs to expire.

@matthew-brett

This comment has been minimized.

Copy link
Contributor Author

commented Jul 25, 2014

I uploaded your wheels here: https://nipy.bic.berkeley.edu/scipy_installers (curl -O from the public URL for your directory, knowing the rest of the URL from the wheel filename). Would you mind uploading to pypi from your rackspace or my copies?

@wiredfool

This comment has been minimized.

Copy link
Member

commented Jul 25, 2014

Uploaded.

@matthew-brett

This comment has been minimized.

Copy link
Contributor Author

commented Jul 25, 2014

Great - thanks

@wiredfool

This comment has been minimized.

Copy link
Member

commented Sep 23, 2014

Closing this -- it's working. Thanks for your help.

@wiredfool wiredfool closed this Sep 23, 2014

@aclark4life

This comment has been minimized.

Copy link
Member

commented Sep 23, 2014

Cool, so we are automatically building wheels… on every Travis run?

@wiredfool

This comment has been minimized.

Copy link
Member

commented Sep 23, 2014

No, it still takes a manual push to that repo to build the OSX wheels, so I'm just doing it on release.

To do it for every build, we'd have to change a setting in the repo to build master/HEAD, change the output filename to something time or commit id based, and setup a trigger that fired off travis on the osx-build repo when there was a change to the main repo.

@aclark4life

This comment has been minimized.

Copy link
Member

commented Sep 23, 2014

Got it, thanks! Still very cool 🍻

@matthew-brett

This comment has been minimized.

Copy link
Contributor Author

commented Sep 24, 2014

No problem - very glad to help - let me know if I can do anything else.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.