Building binaries (wheels) for linux

[gmagno]

Hey hey,

Thanks for this project, this is great stuff!

Have you considered integrating a building step into your travis or gitlab-ci pipeline that would generate Linux binaries (wheels) and push them to PyPI? Something along these lines:

https://github.com/gmagno/pyopencl/blob/master/travis/build-wheels.sh
https://github.com/gmagno/pyopencl/blob/master/.travis.yml

Travis build: https://travis-ci.com/gmagno/pyopencl/builds/98730557

The above repo is currently generating the following builds on a Centos5 which I haven't tested on any other distro besides Ubuntu16.04, but it should work™ on "manylinux"-es:

https://pypi.org/project/pyopencl-gm/#files

This would be important for obvious reasons: not having to build pyopencl and install dependencies (numpy, pybind11 and mako) beforehand, so atm:

pip install pyopencl-gm
python -c "import pyopencl as cl; print(cl.get_platforms())"

just works.

This follows pypa/manylinux (PEP513). They provide an example here: https://github.com/pypa/python-manylinux-demo.

I am fairly new with pyopencl, so I am not 100% confident about the impact on your project.

[kif]

On Sun, 27 Jan 2019 05:04:30 -0800 gmagno ***@***.***> wrote: Hey hey, Thanks for this project, this is great stuff! Have you considered integrating a building step into your travis or gitlab-ci pipeline that would generate Linux binaries (wheels) and push them to PyPI? Something along these lines: https://github.com/gmagno/pyopencl/blob/master/travis/build-wheels.sh https://github.com/gmagno/pyopencl/blob/master/.travis.yml The above repo is currently generating the following builds on a Centos5 which I haven't tested on any other distro besides Ubuntu16.04, but it should work™ on "manylinux"-es: https://pypi.org/project/pyopencl-gm/#files This would be important for obvious reasons: not having to build pyopencl and install dependencies (numpy, pybind11 and mako) beforehand, so atm: pip install pyopencl-gm python -c "import pyopencl as cl; print(cl.get_platforms())" just works. This follows `pypa/manylinux` (PEP513). They provide an example here: https://github.com/pypa/python-manylinux-demo. I am fairly new with pyopencl, so I am not 100% confident about the impact on your project.

Hi Goncalo, The ideal would be to provide, in addition to the OpenCL header and library, also the POCL driver which would actually provide a "ready-to-use" wheel with a portable driver. I have no idea if pocl (v1.2) builds on a manylinux environment. Elder version are rather buggy. Cheers, Jerome

[inducer]

@gmagno I think this is a lovely idea. Thanks for working on this! One main question we should address here is which OpenCL version these should target. I'm guessing 2.0 or 2.1 are the relevant choices. (The ICD installed by Nvidia supports 2.0, but not 2.1.) I'm guessing 2.0 would be my vote for widest compatibility. Would you be willing to create a pull request to add these CI jobs?

Nb: I somewhat disagree with @kif: Yes, given enough effort, we probably could make pyopencl binary wheels that include the entire kitchen sink: an ICD loader (e.g. ocl-icd or the khronos one) and a CPU ICD (e.g. pocl). However that's a lot for a single wheel to do, so I'd prefer we do not do that. Instead, for "just give me a whole functioning setup" type builds, I recommend installing using conda, as described in these instructions. Following those, you go from "I opened a terminal" to "I have a working OpenCL setup" in about four shell commands. (IOW: If we do provide wheels in addition to the conda install procedure, I would imagine the user would still have to have their own ICD loader and ICD.)

[gmagno]

Hey @kif, @inducer, thanks for your feedback.

The ideal would be to provide, in addition to the OpenCL header and library, also the POCL driver which would actually provide a "ready-to-use" wheel with a portable driver.

I see. I am not sure how much effort it would take to build a wheel with pocl in a Centos5. I guess this dockerfile would be a good reference: https://github.com/pocl/pocl/blob/master/tools/docker/RHEL/default

OTOH and as @inducer mentioned, that might be too much for a wheel. In any case this can always be addressed at a later stage if need be.

Thanks for working on this!

You're welcome!

One main question we should address here is which OpenCL version these should target.

I don't yet understand the implications of this in the building process. Please could you elaborate a bit more so I can make the necessary changes to the job. I thought these wheels were compatible with both versions, no?

Would you be willing to create a pull request to add these CI jobs?

@inducer I am more than happy to do that!

Regarding OpenCL-Headers and OpenCL-ICD-Loader, I guess we want to stick to their latest versions. Is this also your understanding?

[isuruf]

I've got some wheels at https://github.com/isuruf/isuruf.github.io/releases/tag/v1.0 with pocl vendored in if anybody is interested.

[gmagno]

That's great!
Any chance you could share how those were built (source)? Not saying I don't trust your binaries, of course... 😄

[inducer]

That's impressive! What would be a reasonable way to make them installable pip install (something) without making them the default?

[isuruf]

What would be a reasonable way to make them installable pip install (something) without making them the default?

We could make a package pyopencl-pocl and change pyopencl so that, pip install pyopencl[pocl] would install both pyopencl and pyopencl-pocl

[isuruf]

And pip install pyopencl would only have the python package and the ICD loader

[inducer]

That's not a bad plan. It would certainly make it easier to adopt PyOpenCL as a dependency for computational work. I'm a little worried about overextending ourselves with maintaining installation paths through conda forge and wheels, but that sort of depends on how finicky the build infrastructure turns out to be. For now, I'm supportive.