Universal binary support for MacOS

[ShaiWavesAudio]

When I pip install PyYAML on a MacOS intel machine I get _yaml.cpython-39-darwin.so as "Mach-O 64-bit bundle x86_64".
When I pip install PyYAML on a MacOS AppleSilicon (arm64) machine I get yaml.cpython-39-darwin.so as "Mach-O 64-bit bundle arm64".

This goes against the established practice of distributing MacOS binaries as "Universal binary" that contains both x86_64 and arm64.

I have a project I distribute with pyinstaller and I rather distribute one universal binary than different ones according to the user's machine type.

Python 3.9/3.10 itself, and all other modules are I used are distributed as Universal binary. pyinstaller also supports building universal binaries.

I there an option to get PyYAML as Universal binary ?

[nitzmahone]

Not shipping universal2 binaries was a conscious choice, since the practical need for most users is pretty low... That does, unfortunately, screw over folks such as yourself that are trying to ship a true multi-arch Python installation.

I don't recall all the problems we had early on that kept us from shipping universal2 wheels- it's possible to construct one manually in multiple passes with lipo, but to use any of the common build tools (we use cibuildwheel), you have to have a libyaml fat binary and arch-neutral headers already available for the single multi-arch build pass build those tools expect. I was working on cffi and pyyaml at the same time, so I don't recall the specifics of the problems for each, but getting a dual-arch libyaml static library + headers was enough trouble that we punted on it and just did arm64. I do recall there were fewer problems with libyaml than there were for libffi, which is still busted upstream. Apple did a major patch to the headers for their libffi build to make it work in their SDK- the upstream libffi still can't be built against in a single pass... I do remember there were non-zero problems to solve for libyaml as well, though.

If you want to try it yourself, just manually build libyaml static objects for both arches, lipo the results together, override the library path to point at the merged library and the header path to pick one arch, then 🤞 that the generated headers for one arch are "close enough" for the other arch that they'll function without segfaulting.

[nitzmahone]

... another thought: if your YAML needs aren't too performance-sensitive, you could just ship a pure-Python installation of pyyaml.

[ShaiWavesAudio]

"Not shipping universal2 binaries was a conscious choice, since the practical need for most users is pretty low"
This goes against the grain of both MacOS and python module development. My project uses at least 10 other modules, all of them have universal binaries. Developing for MacOS can be complected (believe me, I know...) but I'm sure a reasonable solution can be found, as other module authors have done.

My YAML needs are performance-sensitive, but I guess it's pure-Python for now.

Anyway thanks for the details response!

[nitzmahone]

Just for giggles I threw together another pass at this while eating lunch today... I was able to get a single-pass build of upstream libyaml static libs to work (albeit hackily), and a single-pass Python extension build worked under cibuildwheel, but the delocate version that's hardcoded in cibuildwheel bombs while inspecting the extension (probably because it has no dynamic linkage). At a glance, the most recent builds of delocate wouldn't fail in the same way, so when cibuildwheel updates their delocate, this might actually be possible without manual builds.

[rasswanth-s]

@nitzmahone is it possible that we are able to build universal2 binaries for pyyaml currently?
Our project also depends on it.
Would be helpful to know the status