Want any help to build pyo wheels?

[matthew-brett]

Hi - I'm involved in building a lot of wheels for the Python scientific stack - and I was talking to someone who needs pyo wheels for their binary install.

Is there any chance of generating pyo wheels to install via pip / pypi?

Do you want any help doing that? I have lots of experience doing that kind of thing.

[belangeo]

Hi,
I already have wheels for MacOS and Windows (32/64), py 2.7 and 3.5+. Next release will be wheels only. I just can't build wheels for linux because of third-party dependencies. But pyo is packaged for Debian, Fedora and Arch.

[matthew-brett]

Did you come across Manylinux yet? These are wheels that work on almost any Intel Linux. There is machinery to package the dependencies inside the wheel. They have become very popular - almost all big packages have Manylinux wheels now.

I maintain a project multibuild with machinery for building Mac and Manylinux wheels, and vendoring the dependencies. There's another package called scikit-build that I believe can do something similar.

Can I install the Mac wheels from pypi? I tried a week or so ago, with Python 3.7, but that didn't get a wheel.

[belangeo]

But the question is how manylinux can handle portaudio, portmidi, libsndfile and liblo third-party dependencies ? And is it a good idea at all to use these from another source than the one from the current system package manager ?

Can I install the Mac wheels from pypi? I tried a week or so ago, with Python 3.7, but that didn't get a wheel.

Not fully up-to-date but you can probably try the last test I did on test.pipy.org:

https://test.pypi.org/project/pyo/#files

[matthew-brett]

Various other Python packages have a lot of binary library dependencies. The Python Imaging Library - Pillow - is one example. You, and / or me, as the packager, have to build these libraries in the Manylinux docker container. Usually that's not too much work, especially after we have the initial recipe. After that, you build the wheel as usual, then the machinery pulls all the libraries you depend on into the wheel. This does mean you can make sure you have the latest version of all these libraries, and you have complete control of your dependencies.

Of course, if you don't do that, you a) have to build wheels for every possible distribution of Linux, one wheel per release of Ubuntu, Debian, Fedora ... and b) you can't put them on PyPI, because there is no way of specifying which distribution your wheels are valid for, so your user is surprised to find out pip doesn't work for them, and then they have to go searching, to find the URL and set of commands they need to install. This means that other packages will be reluctant to depend on Pyo, because their own installations will break, when pip fails to find a wheel for Pyo.

[matthew-brett]

The procedure that many packages use, is something like Multibuild, where you have a separate repository / Travis-CI job. When you make a commit to the repository, Travi-CI builds and tests the wheels for Mac / Manylinux, then uploads somewhere, usually a Rackspace container, for the packager to fetch, and upload to PyPI, at release time. Examples are Numpy, Scipy, Matplotlib, Pandas, Pillow, h5py and others.

[belangeo]

Ok, I will try when I have some time. It would be very nice to have a pip install on every systems! I'll come back to you if I have questions.

[matthew-brett]

Yes, please do let me know if you think I can help. The Pillow system may be closest to the kind of thing you need:

https://github.com/python-pillow/pillow-wheels

[peircej]

@belangeo it looks like I might soon be forced into providing win64 packages. Above you mentioned that you've built win64 wheels. Are they (or an installer) handy somewhere. Only the 32bit exe files are visible from your downloads

[belangeo]

Hi Jon,
Not online yet but yes, I have wheels for python 2.7-3.5+ for 32-bit and 64-bit on Windows and 32/64-bit on MacOS. Don't know if I'll have the time in the next weeks to work on linux wheels so I think I will release a new version soon with pip installation on windows and MacOS and still rely for some time on package maintainers for linux distros.

[peircej]

Great. I'm basing my standalone distribution on 3.6 64bit now. Having 64bit windows wheels will solve one big issue for me! Thanks!!

[belangeo]

Hi @matthew-brett , any news about the state of manylinux2 (manylinux2010) ? I have created a repo to build wheels for linux (based on the manylinux-demo) but Centos 5 is too old. There is no package for portmidi and liblo and its version of libsndfile misses OGG and FLAC support...

[peircej]

I'm also really keen to hear about this the manylinux2010 rollout so I can build psychtoolbox-python wheels. I've seen the issue at pypa/manylinux#179 but couldn't see a place to test a beta docker environment yet.

[matthew-brett]

Yes, I'm afraid progress is slow. I guess the docker image will be available in a few months, but maybe sooner.

Very few manylinux1 wheels use the system (yum installed) packages, if any. We build all our dependencies on the image, from current-ish source.

[belangeo]

Okay, that confirms what I thought I would have to do! A minimal pyo only depends on libsndfile, so I'll begin with it and keep you informed of developments.

@peircej , if you want to try windows and macos pyo wheels, a testing version is available on testpypi:

https://test.pypi.org/project/pyo/#files

[peircej]

Thanks! I'll take a look!

[belangeo]

A little update... It's almost done! All dependencies are built on the docker and the manylinux wheels uploaded to the releases page of the project https://github.com/belangeo/pyo-linux-wheels .
https://github.com/belangeo/pyo-linux-wheels/releases
There is still some problems, @matthew-brett maybe you can help. When I try the 3.5 wheel on my debian system, some files are not searched at the right place. For instance, when I boot the audio server, I get this message:

ALSA lib conf.c:3558:(snd_config_hooks_call) Cannot open shared library libasound_module_conf_pulse.so (/usr/lib/alsa-lib/libasound_module_conf_pulse.so: libasound_module_conf_pulse.so: cannot open shared object file: No such file or directory)

The file libasound_module_conf_pulse.so exists on my system, but in /usr/lib/x86_64-linux-gnu/alsa-lib, not in /usr/lib/alsa-lib. Pyo does not directly link to alsa-lib, it's portaudio and jackd2 that have alsa-lib as a dependency. Do you know if there is a way to tell binary inside the wheel to search in the system's paths instead of something like an hard-coded path (/usr/lib/alsa-lib is where alsa is installed on centos)?

If I create symlinks in /usr/lib before launching pyo, I get sound with the portaudio backend!

[matthew-brett]

That's odd - auditwheel should detect the indirect dependency and vendor it. Any idea why it isn't detecting libasound_module_conf_pulse.so - is it being loaded dynamically or something?

[matthew-brett]

Sorry - I mean - is libasound_module_conf_pulse.so visible as a dependency via ldd or similar?

[belangeo]

Nope... What I understand from the last few days researchs is that within alsa-lib, there is paths that are set at compile-time (from where the files exists on the system when compiling). On Centos5, it's /usr/lib/alsa-lib/libasound_module_conf_pulse.so while on my debian system it's /usr/lib/x86_64-linux-gnu/alsa-lib/libasound_module_conf_pulse.so.

If I create symlinks from /usr/lib/x86_64-linux-gnu/alsa-lib/ to usr/lib/alsa-lib/, it works.

If I remove .libs/libasound-xxxxxxxx.so and replace it with a symlink to my system's libasound.so, it also works.

I just don't know yet how to make it work out-of-the-box...

[belangeo]

And it's the same for jack. With the binary libjack-xxxxxxxx.so.0.1.0 in .libs, it fails to create a jack client and the process segfault. If I remove and replace it with a symlink to /usr/lib/x86_64-linux-gnu/libjack.so.0.1.0, everything works fine.

[matthew-brett]

Is there any mechanism in the compile settings to link to these libraries statically, or via the usual dynamic library path search, at compile time? It will surely be possible to transplant these libraries into the wheel semi-manually, but it would be much better to work out how to make the standard machinery do that. Is there a mailing list to ask about these issues - and why they do the linking as they do?

[belangeo]

Is there any mechanism in the compile settings to link to these libraries statically

I don't think linking statically an entire audio eco-system is a good idea. Alsa is pretty big, with lot of lib and conf files link to the system it's running...

or via the usual dynamic library path search, at compile time?

That would be great. I think these are automatically included because they don't fit manylinux1 requirements. Nothing in auditwheel to not touch some specified libs?

Is there a mailing list to ask about these issues - and why they do the linking as they do?

I'll ask...

What would be ideal is a post-install hook when installing a wheel. I just tried and running this script, with the same privileges used for the installation) just after the installation fixes all problems:

import os
import pyo

directory = os.path.dirname(pyo.__file__)
libs = os.listdir(os.path.join(directory, ".libs"))

withlibasound = withlibjack = False
libasound = libjack = ""

for lib in libs:
    if "libasound" in lib:
        libasound = os.path.join(directory, ".libs", lib)
        withlibasound = True
    if "libjack" in lib:
        libjack = os.path.join(directory, ".libs", lib)
        withlibjack = True

libasoundfound = libjackfound = False
libasoundpath = libjackpath = ""
for root, dirs, files in os.walk("/usr"):
    for f in files:
        if libasound:
            asound = os.path.split(libasound)[1]
            length = len("libasound")
            if f == asound[:length] + asound[length+9:]:
                libasoundpath = os.path.join(root, f)
                libasoundfound = True
        if libjack:
            ajack = os.path.split(libjack)[1]
            length = len("libjack")
            if f == ajack[:length] + ajack[length+9:]:
                libjackpath = os.path.join(root, f)
                libjackfound = True

        if withlibasound == libasoundfound and withlibjack == libjackfound:
            break; break

if withlibasound and libasoundfound:
    print("Remove %s." % libasound)
    os.remove(libasound)
    print("Create symlink pointing to %s." % libasoundpath)
    os.symlink(libasoundpath, libasound)

if withlibjack and libjackfound:
    print("Remove %s." % libjack)
    os.remove(libjack)
    print("Create symlink pointing to %s." % libjackpath)
    os.symlink(libjackpath, libjack)

[matthew-brett]

We don't need to ship the conf files but the wheel probably does need the sound libraries, to function independently.

There is (deliberately) no way to run a post-install script from a wheel. The only option is to put the post-install into the __init__ of the package. Is that practical? For example, Numpy has something that can be used as a post-install hook with its _distributor_init.py file mechanism: https://github.com/numpy/numpy/blob/master/numpy/__init__.py#L139

All the libraries on the manylinux docker image are compatible, so all the libraries that the Python code links to, should be compatible. I bet though, that some of these libraries are getting dynamically loaded with dlopen, instead of being dynamically linked. If that's true, the question is whether it is possible to configure the build to do dynamic linking instead of dynamic loading.

[belangeo]

The only option is to put the post-install into the __init__ of the package. Is that practical?

I thought about doing this but in most use cases, it needs to be run as root, which becomes a problem as most of the time, the python process is run as user.

[belangeo]

Here we go!

https://test.pypi.org/project/pyo/0.9.3/#files

How it works is on the first run, pyo searches for system's libasound.so and libjack.so and creates symlinks in ~/.pyo. Then, the first thing pyo does when it statrts is to preload those libs with ctypes.CDLL.

It works nicely on my debian computers, but will definitely needs testing on fedora, arch, suse, etc. (with and without jack installed). I'll ask on the mailing list.

[matthew-brett]

Nice! Well done - that's a nasty problem you solved there.

…

On Fri, Apr 26, 2019 at 5:39 PM Olivier Bélanger ***@***.***> wrote: Here we go! https://test.pypi.org/project/pyo/0.9.3/#files How it works is on the first run, pyo searches for system's libasound.so and libjack.so and creates symlinks in ~/.pyo. Then, the first thing pyo does when it statrts is to preload those libs with ctypes.CDLL. It works nicely on my debian computers, but will definitely needs testing on fedora, arch, suse, etc. (with and without jack installed). I'll ask on the mailing list. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#139 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAQQHANI52XQ72CL7BV55DPSMV3JANCNFSM4GTAKUWA> .