macOS: OpenEXR dylib names: changed with each release, breaking dependent binaries

[mascguy]

I'm the MacPorts maintainer for OpenEXR, and we've noticed one minor issue: Each new release uses new dylib names - based on the full release version - breaking dependents binaries. The result being that we must rebuild every dependent.

And while we can live with this if we absolutely have to, it's certainly not ideal. Particularly if the ABI is generally backward-compatible.

Would it be possible to change the naming scheme, such that this doesn't happen with each new release?

p.s. You folks have been absolutely great to work with, and please don't take this as criticism! Just a minor point, which we wanted to bring up.

CC: @JGoldstone

[meshula]

I'm wondering if there's a best practice your team likes? I've been looking at they way different libraries deal with this, and noticed for example that MaterialXGenGlsl.1.38.dylib links MaterialXCore.1.dylib, and the install process installs MaterialXCore.1.38.dylib and a symlink with the abbreviated "major release" name. I was wondering if this is a typical approach to address what you are talking about, or if there are other preferred practices?

[jmroot]

Here's Apple's documentation for library versioning on macOS: https://developer.apple.com/library/archive/documentation/MacOSX/Conceptual/BPFrameworks/Concepts/VersionInformation.html#//apple_ref/doc/uid/20002255-100130

The main relevant point here is that the install_name (AKA the shared library id name) of the library should change if and only if binary compatibility is broken. The install_name is recorded in clients of the library at link time and used to find the library at runtime. It doesn't really matter which way round the symlinks are, as long as the right library exists where the install_name says it should.

You can see the install_name with otool -D. For example, with 3.1.11 we have:

% otool -D /opt/local/lib/libOpenEXR.dylib
/opt/local/lib/libOpenEXR.dylib:
/opt/local/lib/libOpenEXR-3_1.30.dylib

30 is the soversion, and from what I can tell is how openexr tracks binary compatibility. That should definitely be part of the install_name. 3_1 on the other hand looks like it might refer to this being version 3.1.x. That probably shouldn't be in there, because ideally your ABI breaks happen as rarely as possible, so you just increment the soversion when needed and don't assume binary incompatibility based on the software version.

[cary-ilm]

We're happy to work on straightening this out to meet everyone's needs. We made some changes with library naming with the 3.2 release motivated by Linux concerns, but didn't consider the implications of macOS, although they may not be different.

Our ABI changes tend to happen only about once a year, and the patch releases in between should all be compatible, no change to the soversion. So the 3_1 should only change about annually. Are you seeing that our patch releases (e.g. 3.1.x) are triggering downstream rebuilds?

[jmroot]

As far as I know the install_name is only changing on minor releases like 3.1->3.2, not patch releases. (@mascguy please correct if you've observed otherwise.)

I am curious why the ABI is changing every single minor release though. Are you regularly removing legacy interfaces? Or are the semantics of existing interfaces changing in incompatible ways that often?

[cary-ilm]

The ABI-changing minor releases only come once a year or so, with a compatible patch releases every month or so in between. The 3.2 release removed fields from internal data structures and changed some symbols in the ABI, with some minor changes in the API.

[jmroot]

Thanks. So clearly 3.2 did in fact break ABI, but I guess the point I'm wanting to get across is that it would be nice to leave the door open for that to not happen in some minor releases.

[cary-ilm]

That's certainly a possibility, and our versioning scheme does support a minor release that doesn't bump the soversion, although in practice, the two have tended to coincide.

[mascguy]

As far as I know the install_name is only changing on minor releases like 3.1->3.2, not patch releases. (@mascguy please correct if you've observed otherwise.)

Agreed, per my recollection of previous updates.

[lgritz]

There's a nomenclature mismatch here. When the second digit changes (e.g., 3.1 -> 3.2), that is a "major" release for us, no more than annual, and since ABIs are allowed to change at those boundaries, we change the internal namespace and the library name.

Changing first digit, like 3.x -> 4.x, is more major than major, and would be the kind of thing we did not just for ABI changes, but perhaps for totally changing the APIs as well, so user code would need to make different calls entirely and need more than a recompile.

[lgritz]

@jmroot The door is already open: If September rolls around and we don't have anything to release that requires an ABI change, we would just call it 3.2.x+1 instead of forcing a 3.3 where none is needed.

[jmroot]

That's certainly a possibility, and our versioning scheme does support a minor release that doesn't bump the soversion, although in practice, the two have tended to coincide.

OK, if that's the case, then the version number shouldn't be in the install_name, just the soversion. But…

@jmroot The door is already open: If September rolls around and we don't have anything to release that requires an ABI change, we would just call it 3.2.x+1 instead of forcing a 3.3 where none is needed.

If the second number by definition only changes when the ABI changes, I guess there's no problem. :)

[meshula]

OpenEXR has been under rapid development in the last little while. It had been in maintenance mode for years, so maybe it's just surprising to see the numbers ticking up suddenly when for years that hadn't happened :)

[mascguy]

OpenEXR has been under rapid development in the last little while. It had been in maintenance mode for years, so maybe it's just surprising to see the numbers ticking up suddenly when for years that hadn't happened :)

Sounds good. And based on the various discussion, our concerns have been addressed.

Thank you as always folks!