macOS Catalina install fails on insufficient permissions to create volume

[nrdxp]

Describe the bug

Originally reported: nix-community/nix-unstable-installer#13
and: #4994 (comment)

@abathur requested I start a new issue since it is not exactly like #4994

Essentially an install fails on Mac OSX Catalina with the following error:

13>Aug 30 18:59:06 mac-mini-2-signing sudo[619]: getgroupcount called triggering group enumeration
~~> Creating a Nix volume

11>Aug 30 18:59:06 mac-mini-2-signing com.apple.xpc.launchd[1] (com.apple.xpc.launchd.domain.pid.diskutil.625): Failed to bootstrap path: path = /usr/sbin/diskutil, error = 2: No such file or directory
Error starting APFS Volume creation: Insufficient privileges (-69863)

Steps To Reproduce

1. Get a fresh install of OSX Catalina somehow
2. Easiest way to quickly reproduce would be to run numtide/nix-unstable-installer, which only packages the latest version of the install script and overwrites the binary path

Expected behavior

Install should work as it did before #4289

Additional context
We use a pristine image of Mac OSX running as a zfs volume, which is then provisioned with a script which install Nix and some other required stuff (like secrets). I can trivially destroy and reprovision this image as needed for testing.

[nrdxp]

I know you requested a full log @abathur but there really isn't much to report since it fails pretty early on. The only message out of the ordinary is the one shown above.

[abathur]

Most-interested in the log to see how it's invoked; can just edit that in and omit the rest, I guess.

[abathur]

1. Get a fresh install of OSX Catalina somehow
2. Easiest way to quickly reproduce would be to run numtide/nix-unstable-installer, which only packages the latest version of the install script and overwrites the binary path

To clarify a little (mostly for later travellers), this issue is almost certainly narrower than this. A fair number of people have confirmed the installer to be working on ~normal Mac hardware with ~normal nonvirtualized macOS installs, but we've also had a slow trickle of hard-to-debug issues with virtualized/cloud/CI setups. (partial backstory in #4779 and #3156)

~~> Creating a Nix volume

11>Aug 30 18:59:06 mac-mini-2-signing com.apple.xpc.launchd[1] (com.apple.xpc.launchd.domain.pid.diskutil.625): Failed to bootstrap path: path = /usr/sbin/diskutil, error = 2: No such file or directory
Error starting APFS Volume creation: Insufficient privileges (-69863)

My initial guess (because of the Error starting APFS Volume creation: Insufficient privileges: Insufficient privileges bit) was that your case might need the installer to use sudo when it creates the volume, but I hadn't scrolled off to the end of the long log line to see the diskutil error. I stripped off the code block above to make this stand out more.

In a small set of Google search results drilling down on this I see a number of VMWare Fusion users turning up.

I'm curious if you're just seeing a slightly different part of this elephant than others have reported. What does /usr/sbin/diskutil info -plist / | xmllint --xpath "(/plist/dict/key[text()='GlobalPermissionsEnabled'])/following-sibling::*[1]" - print on one of these instances?

[nrdxp]

so the provisioning script is open here:
https://github.com/input-output-hk/ci-ops/blob/master/modules/macs/guest/apply.sh#L66

You'll see here I am using the stable installer, that's because these machines are live and in production so I never committed the failed attempt to upgrade. Just imagine calling numtide/nix-unstable-installer on this line instead

The output of your command is: <true/>.

if I read those two linked issues correctly it's different

[abathur]

True is what we're hoping for, at least. I have a few ideas, but I'll be happy if we can sudo our way through it.

I pushed a test commit to my fork that adds sudo back to the addVolume invocation; you should be able to adapt sh <(curl https://abathur-nix-install-tests.cachix.org/serve/k6c7p3fw0ql1bsyn9ky2c9l1c3rqn622/install) --tarball-url-prefix https://abathur-nix-install-tests.cachix.org/serve to install it (I'm just extracting this from the test run in https://github.com/abathur/nix/actions/runs/1188686737 at https://github.com/abathur/nix/runs/3480531536?check_suite_focus=true#step:4:3).

FWIW, this change cuts a corner for testing; if this works it'll still take a little more brainpower to make a full fix.

[nrdxp]

Thanks for the script, but the curl command above just returns:

{"error":"No such file install"}

[abathur]

Sorry, it's been a few months; grabbed the wrong URL out of the test. I've updated it above (and linked to where the installer URL shows up in the test output).

[nrdxp]

so it looks like we are back to the hash mismatch like the other scripts you sent me:

/Users/nixos/install-nix: SHA-256 hash mismatch in 'https://releases.nixos.org/nix/nix-2.4pre19700101_e6dba53/nix-2.4pre19700101_e6dba53-x86_64-darwin.tar.xz'; expected f80acb1fe64995c695f8b70b2a607229e0623dbaba1e58076351f54a9ff74b06, got b2325624c274158ad212e7d3ac51382278ce1be5cdb50c0e0de4c571f0a91aae

[abathur]

It shouldn't be hitting releases.nixos.org if you're specifying the --tarball-url-prefix.

[nrdxp]

this is how it was invoked:

    curl -L https://abathur-nix-install-tests.cachix.org/serve/k6c7p3fw0ql1bsyn9ky2c9l1c3rqn622/install > ~nixos/install-nix
    sudo -i -H -u nixos -- sh ~nixos/install-nix --daemon --tarball-url-prefix https://abathur-nix-install-tests.cachix.org/serve < /dev/null

[abathur]

It looks like the outer installer only handles --tarball-url-prefix <url> as the first and second arguments; it should work if you'd move --daemon to the end.

[nrdxp]

okay that finally worked. I still saw the errors about /usr/sbin/*: no such file or directory for a couple different files, but it didn't seem to affect the installation.

[abathur]

I still saw the errors about /usr/sbin/*: no such file or directory for a couple different files, but it didn't seem to affect the installation.

These make me a little squeamish, but perhaps they're just noise. Will your next-step be to put these installs through the paces? If so, I'm interested in hearing how that goes. I'll also try to find a little time to try this out on a spare macbook.

[nrdxp]

Yes well I can say that this "noise" never occurred with the stable install script, so it might be something, but so far none of our CI pipelines seem to mind the update on the one machine I'm testing this on atm.

[abathur]

I used a spare macbook running an 11.4 BigSur beta for a fairly basic test of the same installer you tried and didn't have any trouble.

I've opened #5202 with a more-realistic swing at this and tested it on my spare macbook, if you can give it a try as well. The installer url is https://abathur-nix-install-tests.cachix.org/serve/ns7vmm072i37y0zf1nwkvz9v4qzklrga/install this time.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/what-is-the-recommended-installer-to-use-macos/15138/1