macOS (aarch64): Installation immediately errors and can't be undone

[citizen428]

Error

Error: 
   0: Install failure
   1: Error executing action
   2: Action `create_nix_volume` errored
   3: Action `bootstrap_launchctl_service` errored
   4: Failed to execute command with status 119 `"launchctl" "bootstrap" "system" "/Library/LaunchDaemons/org.nixos.darwin-store.plist"`, stdout: 
      stderr: /Library/LaunchDaemons/org.nixos.darwin-store.plist: Service is disabled
      Bootstrap failed: 119: Service is disabled

The offered undo step also fails:

Planner: macos (with default settings)

Planned actions:
* Remove the APFS volume `Nix Store` on `disk3`


Proceed? ([Y]es/[n]o/[e]xplain):
 INFO Revert: Remove directory `/nix/temp-install-dir`
 INFO Revert: Configure Nix daemon related settings with launchctl
 INFO Revert: Configure Nix
 INFO Revert: Provision Nix
 INFO Revert: Create an encrypted APFS volume `Nix Store` for Nix on `disk3` and add it to `/etc/fstab` mounting on `/nix`
Error:
   0: Multiple errors

Location:
   src/cli/subcommand/install.rs:236

Error:
   0: Action `bootstrap_launchctl_service` errored
   1: Failed to execute command with status 113 `"launchctl" "bootout" "system" "/Library/LaunchDaemons/org.nixos.darwin-store.plist"`, stdout:
      stderr: /Library/LaunchDaemons/org.nixos.darwin-store.plist: Could not find specified service
      Boot-out failed: 113: Could not find specified service
      Could not find service.

   1:

Backtrace omitted. Run with RUST_BACKTRACE=1 environment variable to display it.
Run with RUST_BACKTRACE=full to include source snippets.

Additional info

1. /nix was created:

   drwxr-xr-x  - root 20 Jun 23:55 nix
   

2. There's no Nix-related APFS volume that I can see in Disk Utility or with diskutil apfs list
3. Rebooting seems to remove /nix

Metadata

key	value
version	0.8.0
os	macos 13.4
arch	aarch64

[Hoverbear]

Bootstrap failed: 119: Service is disabled

This is suspicious.

Did you perhaps used to have Nix installed?

[citizen428]

A relatively long time ago, from the official installer and I went through a lot of effort to clean up everything after.

Thanks for your help btw!

[Hoverbear]

Can you tell me if /Library/LaunchDaemons/org.nixos.darwin-store.plist exists? According to the original message the service may exist and be disabled. The message on revert seems to contradict that though, quite puzzling...

I suspect you ran this at some point

% sudo launchctl disable system/org.nixos.darwin-store.plist

You may be able to enable the service then bootout it?

% sudo launchctl enable system/org.nixos.darwin-store.plist
% sudo launchctl bootout system /Library/LaunchDaemons/org.nixos.darwin-store.plist

[citizen428]

The file does not exist, sorry, I should have mentioned that earlier.

The bootout fails because the file does not exist:

/Library/LaunchDaemons/org.nixos.darwin-store.plist: No such file or directory
Boot-out failed: 2: No such file or directory

[Hoverbear]

So... is Launchctl failing because we are trying to bootstrap a service which it had previously disabled, but has been removed and entirely recreated? I feel so confused.

[citizen428]

I feel so confused.

I can relate to that, can't quite wrap my head around what's happening here.

[Hoverbear]

When I was poking around I was able to list disabled services (even non-existing ones) via

[64] ephemeraladmin@mac-obliging-starfish> sudo launchctl print-disabled system                                                                              ~

        disabled services = {
                // ...
                "org.nixos.darwin-store.plist" => enabled
                "org.nixos.nix-daemon" => enabled
                // ...
        }

If you run that do you see them disabled?

If so, can you run sudo launchctl enable system/org.nixos.darwin-store.plist?

[abathur]

We've seen this with the official installer at least once (albeit with nix-daemon) in NixOS/nix#6499

[Hoverbear]

If re-enabling the service is indeed the step to take, I think we can write some logic to do that!

[citizen428]

tl;dr: It finally worked. For details on the journey, see below. Thanks @Hoverbear!

---

I re-enabled it but re-running the installer errored out in the exact same way as before. I then also manually removed the service from /var/db/com.apple.xpc.launchd//disabled.plist and rebooted and at least I get further now (it was probably just the reboot that was needed after re-enabling the service, not manually editing the file).

Interestingly, after the reboot, I also had a Nix volume and was asked for a password for it. I removed the volume, and now the installer fails with

The keychain has an existing password for a non-existing "Nix Store" volume on disk disk3, consider removing the password with security delete-generic-password -a "Nix Store" -s "Nix Store" -l "disk3 encryption password" -D "Encrypted volume password"

I already ran the command several times, it assures me the password has been deleted:

❯ security delete-generic-password -a "Nix Store" -s "Nix Store" -l "disk3 encryption password" -D "Encrypted volume password"
keychain: "/Library/Keychains/System.keychain"
version: 256
class: "genp"
attributes:
    0x00000007 <blob>="disk3 encryption password"
    0x00000008 <blob>=<NULL>
    "acct"<blob>="Nix Store"
    "cdat"<timedate>=0x32303233303632343039353331355A00  "20230624095315Z\000"
    "crtr"<uint32>=<NULL>
    "cusi"<sint32>=<NULL>
    "desc"<blob>="Encrypted volume password"
    "gena"<blob>=<NULL>
    "icmt"<blob>="Added automatically by the Nix installer for use by /Library/LaunchDaemons/org.nixos.darwin-store.plist"
    "invi"<sint32>=<NULL>
    "mdat"<timedate>=0x32303233303632343039353331355A00  "20230624095315Z\000"
    "nega"<sint32>=<NULL>
    "prot"<blob>=<NULL>
    "scrp"<sint32>=<NULL>
    "svce"<blob>="Nix Store"
    "type"<uint32>=<NULL>
password has been deleted.

However, the installer still insists that it already exists.

Update: after several tries the password was finally gone for real (in a way the installer liked). I then hit the following problem:

An APFS volume labelled Nix Store does not exist, but there exists an fstab entry for that volume, as well as a service file at /Library/LaunchDaemons/org.nixos.darwin-store.plist. Consider removing the line containing /nix from the /etc/fstab and running rm /Library/LaunchDaemons/org.nixos.darwin-store.plist

So I performed the suggested clean up actions, re-ran the installer and we're back to square 1:

   4: Failed to execute command with status 119 `"launchctl" "bootstrap" "system" "/Library/LaunchDaemons/org.nixos.darwin-store.plist"`, stdout:
      stderr: /Library/LaunchDaemons/org.nixos.darwin-store.plist: Service is disabled
      Bootstrap failed: 119: Service is disabled

Update 2: After this failure I re-enabed the service again. I also removed it from /var/db/com.apple.xpc.launchd//disabled.plist explicitly another time. Deleted the nix APFS volume. Remove the launch script and encryption password from keychain. Then rebooted. After this the installer worked.

[Hoverbear]

Oh my goodness that's such a chore, I'm sorry you had to do all that. :(

The security delete-generic-password problem you faced is one I also faced and don't really know how to deal with it since it appears the able keychain just... responds with success when it clearly didn't happen?

Part of me wonders if we're being too specific with that command somehow...

I'm gonna make a ticket to enable the service if it's disabled, as well as a ticket to revisit the password delete command and see if it's too specific.

Thank you for your cooperation investigating this issue! I'm glad we got it working.

[Hoverbear]

I'm going to close this issue since we have two specific follow ups to address from this. Once those are closed this issue is also fixed of course.

[abathur]

I suspect that the reason the commands are succeeding is that there's more than one credential matching the invocation from previous install attempts.

Edit: I guess it might not be deleting the credential, but I'd rule out duplicates before tilting at that possibility?

Edit edit: Since the official installer doesn't remove creds and this user notes that they used it at some point, they may have had multiple credentials from that timeframe instead of from running the detsys installer.

It's possible this output is out of date since it's from an older macOS, but I do get an error status if there's no matching credential:

$ security delete-generic-password -a "one" -s "two" -l "three" -D "four"; echo $?
security: SecKeychainSearchCopyNext: The specified item could not be found in the keychain.
44

Keeping passwords matched up with volumes is part of why the official shell installer is using the volume UUID in the credential and mounting service definition.