Non-OS utility (lipo) seems required to use the universal biber binary on macOS

[bgvoisin]

On macOS, the first time the universal biber binary is run and its cache is created (as described in §4 Binaries of the doc), the "lipo" utility is called to extract a thinned version of the binary, specific to the architecture on which biber is run (arm64 for me), in the cache.

The problem is, lipo is not part of macOS, it comes with Xcode or the Command Line Tools. For example, on my setup (I have Xcode-beta),

% xcrun -f lipo
/Applications/Xcode-beta.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/lipo

As a result, for a user having only macOS and MacTeX, and nothing else, error messages are returned and a prompt appears, telling lipo is missing and offering to install the Command Line Tools to get it.

This behaviour seems hard-wired in PAR::Packer. It comes from lines 255–321 of

https://metacpan.org/release/RSCHUPP/PAR-Packer-1.056/source/myldr/boot.c

Is it possible to configure things such that the cache creation process is skipped? Or at least, that the cache does not involve thinning the universal binary?

The aim is that a MacTeX user willing to run biber, directly or indirectly, does not have to install additional components. The original report that motivated this issue came from a user typesetting a beamer document in TeXShop: beamer called biber, which called lipo, yielding error messages about a missing lipo in the TeXShop console; the problem was reported as a TeXShop bug, and it took a bit of time to figure out what was happening.

[plk]

You are correct that lipo is called on the universal binary. I was under the impression that this was part of the default MacOS install unfortunately. Initial testing suggested this but perhaps it has moved into the Command Line Tools ...

This is an issue as we have to first split the binary into its thin variants because PAR::Packer interacts with the binary headers in a way that doesn't work with the signing certs in universal binaries (the MacOS biber binaries are signed, as required by TeXLive).

I think that to solve this we would have to speak to the PAR::Packer maintainer and see if this could be done internally there without lipo.

[bgvoisin]

You are correct that lipo is called on the universal binary. I was under the impression that this was part of the default MacOS install unfortunately. Initial testing suggested this but perhaps it has moved into the Command Line Tools ...

What's confusing is that there is indeed a lipo in macOS, at the usual location in /usr/bin:

% which lipo
/usr/bin/lipo

But that's only a "launcher" (or stub, I think Apple calls this a tool shim), launching the lipo binary from the Developer Tools if it is present, and if not triggering the dialog to install the Command Line Tools to get it. That's the same mechanism that led some web sites to recommend running gcc in Terminal to get the Command Line Tools installed.

Once these tools are installed, "xcrun -f" reveals the actual location of the binary:

% xcrun -f lipo
/Library/Developer/CommandLineTools/usr/bin/lipo

If, instead of the Command Line Tools, you have Xcode, you'll get:

% xcrun -f lipo
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/lipo

By contrast, if the binary inside /usr/bin is an actual binary, part of macOS and not requiring the Developer Tools, you'll get the same answer with "which" and "xcrun -f":

% which vi
/usr/bin/vi
% xcrun -f vi
/usr/bin/vi

You can double-check whether the macOS lipo and the Developer Tools lipo are different beasts by running dsymutil -s on them, for example. With /usr/bin/lipo you'll get mention of some tool-shim, and totally different output with the lipo from the Developer Tools.

[plk]

I suspect that whoever I had check lipo was on their generic MacOs install just did a which and saw the stub. I'm not sure what we are going to do about this currently - the workaround is just to install the cmd line tools for now.

[plk]

Please try the DEV universal binary from SF. I have put in a pull request for PAR::Packer for better CLT detection and a better error message including the command to run to install the CLT. This is incorporated into the current DEV release universal binary.

[bgvoisin]

On 13 Nov 2022, at 20:20, plk ***@***.***> wrote: Please try the DEV universal binary from SF. I have put in a pull request for PAR::Packer for better CLT detection and a better error message including the command to run to install the CLT. This is incorporated into the current DEV release universal binary.

After uninstalling Xcode and using "xcode-select -r" to unset any previously set developer directory, here's what I get with the new universal binary: % biber --version xcode-select: error: unable to get active developer directory, use `sudo xcode-select --switch path/to/Xcode.app` to set one (or see `man xcode-select`) biber: Command Line Tools are not installed - run 'xcode-select --install' to install (errno=17) I fear that, with this error message, the user might have the same reaction as the TexShop user who contacted us originally: "Why on Earth am I asked to install the Comnand Line Tools, it's biber that I am running". Perhaps something more explicit would avoid that reaction, like biber: Command Line Tools are not installed (biber requires lipo which comes with them) - run 'xcode-select --install' to install (errno=17) Bruno

[plk]

Please try the DEV universal binary again - I hope that the xcode-select error should be gone. This is still just a workaround as CLT is still required.

[bgvoisin]

This works perfectly: % ./biber --version ./biber: Command Line Tools are not installed - run 'xcode-select --install' to install (errno=0) To be sure to perform the test in the conditions a first-time user would experience, I deleted Xcode, ran "sudo xcode-select --reset", switched off then booted on the Mac again, prior to running the above.

[plk]

Many thanks for testing.

[amunn]

I've just encountered a version of this error on an older Macbook Air that I don't have much installed on. With an up-to-date TeX Live, and MacOS 11.7.4 (the latest OS I can run on this hardware) I get the following:

$ biber
dyld: Library not loaded: /usr/lib/libauto.dylib
  Referenced from: /Applications/Xcode.app/Contents/SharedFrameworks/DebugSymbolsDT.framework/Versions/A/DebugSymbolsDT
  Reason: image not found
lipo: error: unable to locate xcodebuild, please make sure the path to the Xcode folder is set correctly!
lipo: error: You can set the path to the Xcode folder using /usr/bin/xcode-select -switch
biber: extracting x86_64 binary with lipo failed (wstatus=18176)

[bgvoisin]

Not sure what's happening. Have you tried running xcode-select -p which prints the location of the active developer directory? If your setup is OK, this should be something like /Applications/Xcode.app/Contents/Developer.

You can reset things to their default using sudo xcode-select -r or go, in the Xcode app, to Settings (or Preferences, depending on the macOS version) > Locations > Locations and see what's selected in the Command Line Tools menu.

Stack Overflow mentions also

sudo xcode-select -s /Applications/Xcode.app/Contents/Developer

which has the same effect. (Their other recommendations using the Command Line Tools shouldn't be necessary, since you already have Xcode which includes these tools.)

Other than that, maybe a mismatch between your macOS version and Xcode version. Based on the wikipedia, for macOS 11.7.4 Xcode should be version 13.2.1 or below.

[amunn]

Thanks, Bruno. Indeed it was exactly the latter. I had an old incompatible version of XCode installed. Updating to 13.2.1 (thanks for that info!) solved the problem.