Introduce a multi os travis build that builds OSX wheels #247

Bachmann1234 · 2017-04-30T12:30:33Z

Improvements could be made to build the manylinux wheels as well but
this is a big change on its own. When creating a tag the wheel
will pushed as a github release

So this is unlikely ready to merge in its current state but I think its at a point where some feedback would be helpful.

Example build
Example "release"

Here is me sanity testing the 2.7 wheel similarly to how the makefile does

 I   ² venv  ~/Downloads  pip install lxml-3.7.3-cp27-cp27m-macosx_10_11_x86_64.whl
Processing ./lxml-3.7.3-cp27-cp27m-macosx_10_11_x86_64.whl
Installing collected packages: lxml
Successfully installed lxml-3.7.3
 I   ² venv  ~/Downloads  python                                1014ms  Sun Apr 30 08:14:12 2017
Python 2.7.13 (default, Apr  5 2017, 22:17:22)
[GCC 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.38)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import lxml.etree
>>> import lxml.objectify
>>>

and 2.6

 I   ² venv  ~/Downloads  pip install lxml-3.7.3-cp26-cp26m-macosx_10_11_x86_64.whl
DEPRECATION: Python 2.6 is no longer supported by the Python core team, please upgrade your Python. A future version of pip will drop support for Python 2.6
Processing ./lxml-3.7.3-cp26-cp26m-macosx_10_11_x86_64.whl
Installing collected packages: lxml
Successfully installed lxml-3.7.3
 I   ² venv  ~/Downloads  python                                 728ms  Sun Apr 30 08:25:55 2017
Python 2.6.9 (unknown, Apr 30 2017, 08:22:20)
[GCC 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.42)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import lxml.etree
>>> import lxml.objectify
>>>

Some notes:

I dont bother building all the python3 builds for OSX as they will all fail the same way the allowed_failure I do build does. Filed bug here I can help with that bug but not sure what to do next on it.
The linux wheels that get built are likely not useful as they are not manywheels builds. I experimented with doing the manywheels here and while I think its possible I abandoned it (for now) when I hit an FTP error I had no idea how to deal with. here is a Failing build and the abandoned branch

Bachmann1234 · 2017-04-30T12:32:08Z

.travis.yml

 cache:
  directories:
-    - $HOME/.cache/pip
+  - "$HOME/.cache/pip"
+deploy:


This section would have to be changed in master. It currently sends stuff to my repo which obviously wont work if merged.

Basically you update the encrypted api_key to yours (travis can encrypts with the command line client)

If you are feeling brave you can also have travis push to pypi

Bachmann1234 · 2017-04-30T12:32:44Z

travis/travisRun.sh

+python -u setup.py clean
+CFLAGS="-O0 -g" python -u setup.py build_ext --inplace
+CFLAGS="-O0 -g" PYTHONUNBUFFERED=x make test
+python setup.py bdist_wheel


I can update this to only build the wheels if the OS is osx if that makes more sense.

Bachmann1234 · 2017-04-30T12:35:09Z

requirements.txt

@@ -1 +1 @@
-Cython>=0.20
+Cython==0.25.2


This pin is likely not strictly required. I can roll it back if desired.

scoder · 2017-05-01T07:24:42Z

Thanks! I'll look through it and give it a try as soon as I can.

The FTP failure is probably this one:
https://bugs.python.org/issue27973
See this commit: d0125dc

Bachmann1234 · 2017-05-01T11:16:41Z

.travis.yml

 cache:
  directories:
-    - $HOME/.cache/pip
+  - "$HOME/.cache/pip"


TODO: probably a good idea to add "$HOME/.pyenv" to this

Bachmann1234 · 2017-05-01T12:16:49Z

Think I realized what I was doing wrong for manywheels. Ill update this PR assuming I did it right.

https://travis-ci.org/Bachmann1234/lxml/builds/227574774

https://github.com/Bachmann1234/lxml/tree/linux_manywheels

Bachmann1234 · 2017-05-01T13:31:57Z

Bah! So close. Gets all the way to "show wheels" and explodes due to a missing directory. I'll have that a deeper look tomorrow

anthrotype · 2017-05-01T17:17:50Z

I suggest you also have a look at https://github.com/matthew-brett/multibuild

It makes it super easy to build manylinux1 and osx wheels with Travis CI.

Bachmann1234 · 2017-05-01T18:06:20Z

I'll check it out. Thanks!

Bachmann1234 · 2017-05-02T12:32:08Z

I went ahead and added the linux wheels since I have a working build

https://travis-ci.org/Bachmann1234/lxml/builds/227923232

Still working out a kink... the log shows the wheels being made the the linux wheels have not uploaded.

Bachmann1234 · 2017-05-02T13:28:59Z

Solved the last problem during the commute https://github.com/Bachmann1234/lxml/releases/tag/20170502-5

I'll rebase later

Bachmann1234 · 2017-05-04T12:20:17Z

as you may have noticed im still iterating on this. Im trying to enable static osx wheels. Im doing this by fixing the download code which fails in the osx environment. Ill post the tag if the build passes (I think it will but im wrong a whole bunch)

Improvements could be made to build the manylinux wheels as well but this is a big change on its own. When creating a tag the wheel will pushed as a github release

I realized io is just docker linking to the repo

Bachmann1234 · 2017-05-04T12:38:18Z

Ugh, Spoke too soon. One of the downloads timed out. Ill rebase and push to see if thats a blip. Will also resolve the conflicts

Bachmann1234 · 2017-05-04T13:14:52Z

That time it worked
https://travis-ci.org/Bachmann1234/lxml/builds/228720176
https://github.com/Bachmann1234/lxml/releases/tag/20170504-1

Think im done fooling with this PR until you get a chance to look at it. I managed to make a python3 wheel on osx but that wheel fails the tests with "Unsupported encoding UCS-4LE"

scoder

The previous use of ftplib was intentionally removed here: a882fd9
There were, admittedly, some issues with that, but proxy usage is a nice feature by itself.

Bachmann1234 · 2017-05-05T10:42:27Z

@scoder aw jeez. OK I guess ill have to fallback to ftplib if the http call fails

…ttperror

scoder · 2017-05-07T14:14:28Z

Looks cool. Thank you for going through all this. Let's see what happens when I merge it. :)

Bachmann1234 · 2017-05-07T14:40:23Z

No problem! Just remember to redo the depoly section with your repo. Travis has a client that should do it for you

anthrotype · 2017-05-07T15:08:04Z

@Bachmann1234 thanks for working on this!
I see one problem though. You are using a python built with pyenv for building the osx wheels. This is not good, because since that python has been built with pyenv for that target machine (10.11 sdk, 64 bit architecture), the wheels generated from it are also going to work only on that platform (or above). It is recommended when building wheels for mac os x, to use the Python.org installers for OSX which are compiled as "fat" binaries (both 32 and 64 bit) and using the 10.6 as minimum deployment target. This ensures that the produced wheels will work anywere, and not only on latest OS versions, or with 64-bit-only python builds.

The multibuild tools that I recommended above uses the Python.org's builds:
https://github.com/matthew-brett/multibuild/blob/master/osx_utils.sh

All the osx wheels for the python scientific packages are being built using multibuild now, e.g. see https://github.com/MacPython

Bachmann1234 · 2017-05-07T15:15:19Z

@anthrotype Alrighty, ill take a look sometime next week most likey!

anthrotype · 2017-05-07T15:18:55Z

Oh, I just found this: https://github.com/joerick/cibuildwheel
I haven't tried, but looks even easier than multibuild to set up.
It seems to be related to multibuild, as they mention its author Matthew Brett in the credits section.

Bachmann1234 · 2017-05-07T15:27:18Z

Hmm. I've been hesitant to touch the windows stuff cuse the scope of this work had already grown. But what the heck, I'll give it a shot. If this works the whole system will be simpler

Bachmann1234 · 2017-05-07T15:28:10Z

That will be a different branch/pr

Bachmann1234 · 2017-05-09T02:00:16Z

Welp. My laptop decided to give out on me so I won't be able to look at that fancy build until I get that sorted. Probably at least a week

Bachmann1234 · 2017-05-09T02:40:40Z

Ugh, looks like its gonna be more like 2 weeks.

Bachmann1234 · 2017-05-19T11:14:51Z

Ok, im back. Let me take a look at this thing @anthrotype posted

Bachmann1234 · 2017-05-19T12:21:19Z

Ok, I played with both I think using multibuild is worth exploring. I'm trying that out here https://github.com/Bachmann1234/lxml-wheels

Its not anywhere near done. But ill keep playing with it.

@scoder is this an approach you would be ok with? As soon as I got this working I could hand the repo over. The other option would be to keep hacking away at this one and fix the issues @anthrotype mentioned.

scoder · 2017-05-19T12:35:01Z

Seems much better, if only because it does not depend on specific CPython version numbers. The git subrepo also seems (not great but) ok. Updating the sub-repo link for each release seems doable. And I like the fact that it's a separate repo with a separate travis.yml file, so that the normal lxml repo can continue to do simple CI test builds.

Bachmann1234 · 2017-05-21T15:18:42Z

Ok, think I got this working.

https://github.com/Bachmann1234/lxml-wheels/releases/tag/wheel-test-4
https://travis-ci.org/Bachmann1234/lxml-wheels/builds/234419129

This should be fairly easy to use. We just update the BUILD_COMMIT for each release in .travis.yml

Some notes:

I have not done windows yet but that should be fairly easy (and you probably dont need it since you have that automated... but eh, one repo for all the wheels could be nice)
Im building python 3 wheels of OSX even though https://bugs.launchpad.net/lxml/+bug/1687236 suggest these wheels wont quite work (though neither will pulling from pypi and building it yourself so I guess its still an improvement)
No python 2.6, 3.3 mac wheels. I filed https://github.com/matthew-brett/multibuild/issues/30

Bachmann1234 · 2017-05-21T15:25:44Z

Should I close this and we continue talking in the bug form? What are next steps?

Bachmann1234 · 2017-05-31T23:08:12Z

Just an update. I have 3.3 mac wheels. But I wont be doing 2.6 mac wheels since to do so I would have to avoid ssl validation by downgrading pip (pypa/pip#829) which I don't see as worth it (security concern for an ancient python version and users who really want that can build their own wheel)

I dont think I have anything else to do. Just poke me if there is a next step. Repo is here. https://github.com/Bachmann1234/lxml-wheels

if there is interest in using this repo ill write some notes in the readme

scoder · 2017-06-01T06:18:33Z

Sorry for being slow in responding and thank you for your continued investment into this. I'll give it another try this weekend. Still have to get the travis upload set up on my side.

scoder · 2017-06-12T14:26:11Z

I forked your repo into the lxml github organisatiion.
https://github.com/lxml/lxml-wheels
It nicely builds your lxml branch on travis:
https://travis-ci.org/lxml/lxml-wheels/builds/241921367
But master fails for 2.7:
https://travis-ci.org/lxml/lxml-wheels/builds/241999467
So I think it's missing that FTP fix. Could you maybe extract that as a separate pull request?
I uploaded the wheels it built, let's hope they work for everyone.
https://pypi.python.org/pypi/lxml/3.8.0
Thank you for your efforts!

Bachmann1234 · 2017-06-12T14:27:27Z

I already did #249

On Mon, Jun 12, 2017 at 10:26 AM scoder ***@***.***> wrote: I forked your repo into the lxml github organisatiion. https://github.com/lxml/lxml-wheels It nicely builds your lxml branch on travis: https://travis-ci.org/lxml/lxml-wheels/builds/241921367 But master fails for 2.7: https://travis-ci.org/lxml/lxml-wheels/builds/241999467 So I think it's missing that FTP fix. Could you maybe extract that as a separate pull request? I uploaded the wheels it built, let's hope they work for everyone. https://pypi.python.org/pypi/lxml/3.8.0 Thank you for your efforts! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#247 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABc1xXylMAhokV-h2QGr9MoiEUb5237xks5sDUqIgaJpZM4NMjkJ> .

-- -Bachmann

scoder · 2017-06-12T14:38:59Z

Ah, sorry, had forgotten about that one. There's always a next release ;)

Bachmann1234 · 2017-06-12T14:49:02Z

👍 Thanks. Let me know if there are any issues :-D

Bachmann1234 commented Apr 30, 2017

View reviewed changes

Bachmann1234 commented May 1, 2017

View reviewed changes

Bachmann1234 and others added 4 commits May 4, 2017 08:21

Introduce a multi os travis build that builds OSX wheels

0a64460

Improvements could be made to build the manylinux wheels as well but this is a big change on its own. When creating a tag the wheel will pushed as a github release

Taking a shot at building manywheels on travisci

5fd60a6

Don't make an io dir silly

df4bc7c

Update .travis.yml

bf6def6

I realized io is just docker linking to the repo

Bachmann1234 force-pushed the build_wheels branch from 9efc6d2 to 55d5e59 Compare May 4, 2017 12:42

scoder reviewed May 5, 2017

View reviewed changes

Enable OSX static wheels by falling back to ftplib if urllib has an h…

52f271d

…ttperror

Bachmann1234 force-pushed the build_wheels branch from 55d5e59 to 52f271d Compare May 5, 2017 11:05

stephengroat mentioned this pull request May 12, 2017

add sample osx build #248

Closed

Bachmann1234 mentioned this pull request May 20, 2017

Use ftplib when urllib fails #249

Merged

scoder closed this Jun 12, 2017

		@@ -1 +1 @@
		Cython>=0.20
		Cython==0.25.2

Introduce a multi os travis build that builds OSX wheels #247

Introduce a multi os travis build that builds OSX wheels #247

Conversation

Bachmann1234 commented Apr 30, 2017 • edited

Bachmann1234 Apr 30, 2017

Choose a reason for hiding this comment

Bachmann1234 Apr 30, 2017

Choose a reason for hiding this comment

Bachmann1234 Apr 30, 2017

Choose a reason for hiding this comment

scoder commented May 1, 2017

Bachmann1234 May 1, 2017

Choose a reason for hiding this comment

Bachmann1234 commented May 1, 2017

Bachmann1234 commented May 1, 2017

anthrotype commented May 1, 2017

Bachmann1234 commented May 1, 2017

Bachmann1234 commented May 2, 2017

Bachmann1234 commented May 2, 2017

Bachmann1234 commented May 4, 2017

Bachmann1234 commented May 4, 2017

Bachmann1234 commented May 4, 2017 • edited

scoder left a comment

Choose a reason for hiding this comment

Bachmann1234 commented May 5, 2017

scoder commented May 7, 2017

Bachmann1234 commented May 7, 2017

anthrotype commented May 7, 2017 • edited

Bachmann1234 commented May 7, 2017

anthrotype commented May 7, 2017

Bachmann1234 commented May 7, 2017

Bachmann1234 commented May 7, 2017

Bachmann1234 commented May 9, 2017

Bachmann1234 commented May 9, 2017

Bachmann1234 commented May 19, 2017

Bachmann1234 commented May 19, 2017

scoder commented May 19, 2017

Bachmann1234 commented May 21, 2017

Bachmann1234 commented May 21, 2017

Bachmann1234 commented May 31, 2017 • edited

scoder commented Jun 1, 2017

scoder commented Jun 12, 2017

Bachmann1234 commented Jun 12, 2017 via email

scoder commented Jun 12, 2017

Bachmann1234 commented Jun 12, 2017

Bachmann1234 commented Apr 30, 2017 •

edited

Bachmann1234 commented May 4, 2017 •

edited

anthrotype commented May 7, 2017 •

edited

Bachmann1234 commented May 31, 2017 •

edited