Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Building wheels #70

Open
jakirkham opened this Issue Mar 3, 2018 · 10 comments

Comments

Projects
None yet
3 participants
@jakirkham
Copy link
Member

jakirkham commented Mar 3, 2018

As numcodecs includes source that needs to be compiled, there can be some challenges or technical hurtles that can be encountered by a user that they may not be aware of. While we do solve this in some sense by supplying conda-forge packages with prebuilt binaries, pip remains the defacto way for Python users to get packages. However in the case of pip the user will get the sdist, which needs compilation. While this does work, there is definitely some appeal to providing a pip solution that does not require compilation (i.e. prebuilt wheels). This would help users avoid compatibility problems like this one ( #69 ).

One solution would be to piggy back off of whatever conda-forge ends up doing to also supply wheels ( conda-forge/conda-smithy#608 ). This would work well for Windows. Though on macOS conda-forge uses the 10.9 SDK (instead of the 10.6 SDK that Python tries to support). Also on Linux conda-forge uses CentOS 6 with glibc 2.12 (instead of CentOS 5 with glibc 2.5 used by manylinux1). In practice these two cases that don't sync up will be hard for Python to require much longer and some packages already don't comply (happy to go into details if this is of interest). This certainly would enjoy the nice benefit of the architecture and community conda-forge has in place to solve these issues. It also seems that proponents of the wheel format are interested in collaborating, which definitely should help.

If that doesn't work, the alternative would be to build wheels here. For Linux, there is manylinux1 Docker image, which could be used fairly easily to build these. For macOS, we could try to build them here or reach out to MacPython for help. For Windows, it would probably be best to reuse the conda-forge solution as much as possible as that already fits the requirements well. Probably would require some like issue ( conda/conda-build#2490 ) to solved. Though I don't think anyone that has had time to do that. Alternatively one can build a wheel with conda-build.

@jakirkham

This comment has been minimized.

Copy link
Member Author

jakirkham commented Mar 5, 2018

Have a couple of examples of wheel builds in this comment. For Numcodecs, in particular, please take a look at PR ( conda-forge/numcodecs-feedstock#17 ), which would build both conda and wheel packages from the same build.

@alimanfoo

This comment has been minimized.

Copy link
Member

alimanfoo commented Mar 5, 2018

@jakirkham

This comment has been minimized.

Copy link
Member Author

jakirkham commented Mar 5, 2018

Thanks @alimanfoo.

Right now my feeling is reusing our work at conda-forge to do this is the best path. We will have more resources generally (compute, manpower, storage, etc.) to push this forward. Not to mention we can easily store the wheels on anaconda.org right along side our conda packages.

Expect the wheel folks will be interested in smoothing out the compatibility issues once they see some demonstrated success of wheels in conda-forge. Perhaps by making manylinux2 for instance or broadening the manylinux spec to include arbitrary glibc. There's already some pressure on them to do this as tensorflow and other packages are breaking the manylinux spec anyways ( tensorflow/tensorflow#8802 ). Also CentOS 5 is an unsupported OS as of last year. So they will have to update and they know this ( pypa/manylinux#96 ).

The technical issues of getting wheels in there are minimal IMHO. Will have to see how others in conda-forge feel about having wheels included.

@jakirkham

This comment has been minimized.

Copy link
Member Author

jakirkham commented Mar 5, 2018

@jeromekelleher

This comment has been minimized.

Copy link
Contributor

jeromekelleher commented Apr 25, 2018

Just wanted to say that binary wheels on PyPI would be useful for me. I like to test my code using both the pip and conda infrastructure. Compiling numcodecs for each CI build takes quite a bit of time and a bdist_wheel would be very handy.

I'm loving zarr/numcodecs by the way; thanks for all the work!

@alimanfoo

This comment has been minimized.

Copy link
Member

alimanfoo commented Apr 26, 2018

@jakirkham

This comment has been minimized.

Copy link
Member Author

jakirkham commented Apr 27, 2018

Well we are in the midst of migrating to conda-build 3. So that is pretty much zapping all developer time. Not to mention other work activities outside of conda-forge. IOW it is stalled due to lack of time.

That said, we are able to build wheels using the conda-forge recipe using PR ( conda-forge/numcodecs-feedstock#17 ). There are of course the compatibility issues we already discussed mainly on Linux, but it looks like PyPA is working on a new manylinux2 spec ( pypa/manylinux#152 ), which would resolve this. The main issue is uploading the wheels actually. The current upload script would need to be reworked for wheels ( conda-forge/conda-forge-ci-setup-feedstock#6 ). Not sure what that entails yet.

@alimanfoo

This comment has been minimized.

Copy link
Member

alimanfoo commented Apr 30, 2018

Thanks @jakirkham. Am I right in understanding that conda-forge/numcodecs-feedstock#17 is now successfully building wheels on all platforms?

@jakirkham

This comment has been minimized.

Copy link
Member Author

jakirkham commented Apr 30, 2018

It does appear that way. Hadn't had time to check since that comment. :)

The main challenge is we don't have a way to upload them from the CIs yet. Though one could certainly build those locally and upload them to PyPI or wherever one wishes.

@alimanfoo

This comment has been minimized.

Copy link
Member

alimanfoo commented May 1, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.