darwin: encrypt nix volume if filevault is enabled

[abathur]

This refactors WIP from #4181 to address things that became obvious:

1. The macOS install is getting too complex for single-user mode.
2. There's already frustration around difficulty of uninstalling multi-user on macOS.
3. I want to support upgrading existing macOS volumes to work the same as those made by this WIP, but simply tacking it on as an extra option at all relevant points was looking like a lot of extra conditional logic and calls to some fairly slow diskutils commands.
4. The greater number of Nix-related artifacts shook loose additional concern around avoiding state/artifact inconsistencies.
5. As I developed a curing process to address these inconsistencies, I realized it won't take too much more work to extend these concepts to uninstalling the other Darwin and general Nix multi-user artifacts. I'm pleasantly surprised how practical an uninstall mode feels from here.

The history had grown unruly even before this refactor, so for now I'm flattening this into an omnibus commit to start fresh. Since this touches on a lot, I've tried to better-document the high-altitude view by making three outlines:

• the general process/logic in pseudocode (as-is, and for the uninstaller I'm imagining) removed this for merge
• how high-level user-facing functionality changes here
• what lower-level implementation changes I'm making

High-level functionality changes

• disable single-user installs on macOS
• if the user's root is read-only, create a volume (overridable default)
  • --darwin-use-unencrypted-nix-store-volume flag is now a no-op warning
  • if the boot volume has FileVault enabled, encrypt this volume
    • generate a random credential and place it in the system keychain
  • create a LaunchDaemon to mount it
    • still use fstab to specify mount opts (rw,noauto,nobrowse,suid,owners)
      • noauto so that the system won't auto-mount if our mounter breaks (goal: the user finds/presents for help)
    • the daemon isn't strictly necessary for unencrypted volumes, but I'm using it in both cases for consistency
• update volume naming/customization
  • default: use or create volume named "Nix Store" on same disk as /
    • I had changed this to "Nix", but I want to support up-converting current "Nix Store" volumes and can't justify the complexity of supporting both names.
    • no longer tries to match and use volumes labeled [nN]ix.* (see [OSX catalina] install script fails if a volume starting with [nN]ix.* already exists #4150)
  • overridable variables for the filesystem, label, and target disk
  • overridable variables to BYOV: volume special device, and UUID
• support curing any volume-related state (if present) on install
  • if Nix Store is encrypted
    • verify we can unlock it from keychain
    • otherwise, prompt the user to delete it or manually add the pw to keychain
  • if filevault is on but Nix Store isn't encrypted, offer to encrypt it
  • if nix is the only item in fstab and synthetic.conf, remove them; otherwise suggest edits that will remove the nix line
  • remove the mounter launchdaemon

Lower-level implementation changes

• scaffold uninstall and combined uninstall+install actions
• multi-user installer
  • add reminder/remind UI/X idiom (add notes as we go, show all at end)
    • no more saving variables or re-testing conditions at the end
    • if interactive: show one at a time and require acknowledgement
    • convert some existing messages to use this
  • add password_confirm UI/X idiom for actions like deleting a volume
    • forces a sudo prompt (even when headless or sudo was recently authed)
  • add confirm_rm and confirm_edit UI/X idioms
    • these are "for" curing volume state, but I anticipate them being useful for uninstaller (and, I guess, for the installer if we wanted to be more interactive/explicit around some edits)
  • try to make create_directories more idempotent (may cause trouble?)
    • if /nix already exists, chown -R root:nixbld it
    • use install -dv -m instead of mkdir -pv -m
    • combine dir-structure steps 2 & 3 with install's -g (group) flag
  • only defer to a pre-set NIX_SSL_CERT_FILE if its physical path is not in /nix/store
    • multi-user reinstall over a single-user install was skipping cert setup
  • fail earlier (before sudo lecture etc.) if EUID=0
  • add two new install phases:
    • (poly_)cure_artifacts for reverting changes Nix makes
    • poly_prepare_to_install for preparing the Nix volume on macOS
  • normalize SCRATCH/mktemp path when macOS /usr/bin/mktemp is used:
    • /usr/bin/mktemp -d -t tmp.XXXXXXXXXX --> /.../tmp.XXXXXXXXXX.2RMrVwkx
    • /usr/bin/mktemp -d "${TMPDIR:-/tmp/}tmp.XXXXXXXXXX" --> /.../tmp.JwmZDxN1kf
  • _sudo "explanation" command ... wrapper now sends explanation to stderr
    • make it easier to pipe commands using the wrapper; see create-darwin-volume.sh:volume_pass_works() for an example
• refactor create-darwin-volume.sh to
  • be sourceable as a library but still support direct invocation
    • use UI/X idioms from the multi-user installer
    • give multi-user installer fine-grained control of what runs when
    • ~legacy (bare) direct invocation
      • in case we have to reinstate single-user
      • avoid no/low-notice breaks for anyone who has been directly running it
    • externally invoke a non-main func with args "no-main" "funcname" ...
    • I've seen a fair number of modified forks of this script; over time I hope sourceability and external function invocation will make it easier to re-use
  • largely eradicate a pattern, <run very slow diskutil command> | <parse a single bit of info out with an xpath query>, which is OK in isolation but was becoming a bit too common
  • remove (most) instruction dumps that describe how to remove things that the curing process can now handle
  • defensively use absolute default paths for some common executables (coreutils, find, grep, etc.)

@LnL7 @lilyball @matthewbauer @dhess @mroi @domenkozar (others I'm forgetting are inevitably also interested; feel free to add people)

Try it out

I have been hosting my own installers for much of the history of this PR. Nix now has a better installer-testing process, so I'm going to see if we can lean on it here. I'll leave the old process note here for a bit in case there are problems with the new approach. If you can't try it out, you can see output from the CI run.

The new approach requires an extra flag, so it'll be easiest to just use in interactive mode:

sh <(curl https://abathur-nix-install-tests.cachix.org/serve/yihf8zbs0jwph2rs9qfh80dnilijxdi2/install) --tarball-url-prefix https://abathur-nix-install-tests.cachix.org/serve

Here's a history of the versions it has pointed at, with the current at the bottom (note: I'll only update this for substantive changes or breakage):

• 6132132 (equivalent to 401afa0, but without the reference darwin_volume_reference.py)
• 6bb3bc9
• 0bd1083
• 86dbee7
• 97269f6
• c235bed
• eab14a6

old privately-hosted installer (out of date--use only if above doesn't work)

I have a temporary installer up at https://files.t-ravis.com/install-multi, so curl https://files.t-ravis.com/install-multi | sh (headless) and sh <(curl https://files.t-ravis.com/install-multi) (interactive) should work.

Other issues

I believe this PR:

• closes error: could not set permissions on '/nix/var/nix/profiles/per-user' to 755: Operation not permitted #3435 (but see my comment there)
• closes Multi-User Darwin Installer: No side effects before old installation check #3667
• obsoletes Don't stop user from installing the nix store on an unencrypted volume #3692
• closes [OSX catalina] install script fails if a volume starting with [nN]ix.* already exists #4150
• closes Mac installation should not rely on T2 chip encryption #4380
• closes Reinstalling on macos: "/nix: Read-only file system" #4390
• closes Big Sur 11.2 diskutil: command not found #4488
• closes Jeeze, Install on MacOS Big Sur went wrong. #4666
• closes Nix install failed on macOS 11.2.3 with FileVault and a pre-existing unencrypted Nix Store volume #4675

[dhess]

Thanks once again for this work and the attention to detail.

Does anyone know: on T2 Macs, if you encrypt a user-created APFS volume, is the encryption/decryption for that volume still done on the T2, as with the system volumes; or is it done in software on the CPU?

[abathur]

on T2 Macs, if you encrypt a user-created APFS volume, is the encryption/decryption for that volume still done on the T2, as with the system volumes; or is it done in software on the CPU?

I think it's on the T2, but I won't swear I personally know this as a fact. For example, see the last line below:

$ diskutil apfs encryptVolume disk1s7 -user disk -passphrase foo
Encrypting with the new "Disk" crypto user on disk1s7
The new "Disk" user will be the only one who has initial access to disk1s7
The new APFS crypto user UUID will be 920D4EAD-BFAE-457B-9A19-827AAC0C1B42
Encryption has likely completed due to AES hardware; see "diskutil apfs list"

If you encrypt a volume on a pre-T2 mac, that line will say Background encryption is ongoing; see "diskutil apfs list" to see progress

[LnL7]

That's also how I understood the T2 documentation from apple https://www.apple.com/jp/mac/docs/Apple_T2_Security_Chip_Overview.pdf.

The chip handles all the encryption and is actually always enabled, FileVault only changes the conditions for which the chip unlocks the data. Meaning that the data also doesn't have to change when updating credentials. Which the behaviour you noticed seems to confirm is the case.

[abathur]

In case anyone's curious, I added screenshots of the ~interactive UI/X elements in a comment alongside the code for each (and immediately resolved them, so they don't take up gobs of space in the main thread).

[lilyball]

For future reference, you can use <details> tags to hide screenshots by default.

<details><summary>Description of screenshot</summary>

// drag image into this blank line, if it's surrounded by blank lines the markdown ![image](url) syntax will work just fine.

</details>

This will show up as

Collapsed summary here

Once expanded the image will be visible.

[lilyball]

This is only a partial review, I’ll have to come back to this later.

[abathur]

I've pushed commits addressing one round of comments from @lilyball and @SuperSandro2000 (also thanks to @samueldr and @jtojnar for additional discussion). Going to go ahead and convert this from a draft.

I've started a build of this locally, and will update to confirm its status (status: my install checks are working as of 6bb3bc9, but the Nix repo checks failed at this commit--I'd be surprised if this caused that, so I assume it's a transient failure or flakiness?)

For anyone champing at the bit, I suspect we'll be holding up just a little bit (if it's a short wait) for the chance I'll be able to help @domenkozar test a new GitHub Actions integration per #4047 that should make it easier to test PRs that touch the installer.

In any case, reviews still welcome in the meantime. :)

[lilyball]

Another partial review. Apologies for not managing to do a full one.

[lilyball]

I'm not sure if this really matters, but maybe we should stick with /bin/bash just to ensure we're using the exact same version of bash everywhere?

[abathur]

Good question.

I just adopted the shebang from install-multi-user.sh for consistency.

I'm not sure how to weight consistency with install-multi-user against being explicit here. Everything install-multi-user.sh sources has to be 3.2 compatible for this reason (and I did a lot of manual 3.2 testing across all of the scripts as I went).

I assumed/inferred from the #!/usr/bin/env bash shebang on install-multi-user that at least some of the platforms we might be run on don't have a /bin/bash, but if that's not true I suppose they could all change.

What do you think?

[lilyball]

I suppose consistency is worthwhile. If we do need to run install-multi-user.sh on a platform that doesn't have /bin/bash then the current shebang makes sense.

[abathur]

bf5290d fixes an oversight caught by lilyball regarding curing/checking for the presence of Nix as a symlink in synthetic.conf. You can see sample output in this CI run:

• success on symlink in synthetic.conf, but not stitched
• fail on symlink in synthetic.conf stitched

The stitched jobs fail because once we've stitched, we need a reboot to remove the symlink.

[ethnt]

@abathur Thanks a lot for your work on this! Is there an ETA for when this will be made publicly available?

[abathur]

Not sure. We're on the crowded side of a bottleneck. :)

[grahamc]

I've seen good success with this PR and I think it is time to give our users a better experience, so I'm going to go ahead and merge it :)

[jgeerds]

@abathur Thank you very much for your work! Am I right that these changes only affect nixUnstable and upcoming releases? If so, is there a chance for a 2.3.x backport?

[abathur]

@jgeerds Yep.

Backport decisions are above my paygrade, but unless 2.4 is impending I think it spares enough pain to be worth a backport and release.

[abathur]

This may also be a good place to note: I think the merge of this PR lights a path to full uninstall/reinstall. I can't pursue it any time soon, but I'm happy to discuss as needed if someone else can. I wrote a proof-of-concept in Oct. that should be reheatable. Validation is also much easier since domen added installer tests.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nix-macos-monthly/12330/2

[rpearce]

Holy smokes; what a great install experience. This worked for me on a macOS early 2013 running 10.15.7. Thank you!!!

[rpearce]

@abathur Sorry to bother: how will we know when this will become the default installer for Darwin? This is what I'm telling coworkers currently:

sh <(curl https://abathur-nix-install-tests.cachix.org/serve/yihf8zbs0jwph2rs9qfh80dnilijxdi2/install) --tarball-url-prefix https://abathur-nix-install-tests.cachix.org/serve

[abathur]

@rpearce No worries. Subscribing to this thread is probably the best bet.

I deferred a sprawling, intricate refactor of another project for a few months to focus on this PR. For my own sanity, I had to stick a fork in this a couple months back so that I could shift enough of my focus back to that effort to make any headway.

I guess we're waiting for either:

• a new release from master (whether that's 2.4 or 3.0)
• someone to drive an effort to backport it to the 2.3-maintenance branch and get a new release from there

It can be backported (and I agree with releasing it), but there are a few caveats that probably make it more complicated than just bugging domen:

1. There are other commits in master that haven't been backported, and this PR is extensive enough that I imagine someone would need to audit the other changes there to make sure anything this builds on (and any fixes to this?) get pulled in. A good place to start is probably something like: git log --color=always --format='%C(auto,blue)%h%Creset %C(auto,green)%cs%Creset [%C(auto,red)%<|(37)%S%Creset]%C(auto,yellow) %m %Creset%s' --cherry-mark --no-merges master...2.3-maintenance -- scripts | grep -v ' = '
2. The documentation format was converted from xml to markdown between 2.3-maintenance and this work, so the doc changes will need to be re-applied in the xml.
3. The installer-testing work domen implemented this winter isn't present in 2.3-maintenance, so someone will need to either backport that work as well, or manually generate installers to test them (i.e., the old way :/)

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/anyone-up-for-picking-at-some-nix-onboarding-improvements/13152/1

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/how-to-install-nix-on-macos-with-m1/13791/4

[mpscholten]

An issue was just reported to IHP with the following error running this installer:

SHA-256 hash mismatch in 'https://abathur-nix-install-tests.cachix.org/serve/jqr52m6w8dwag5932hw67kq0rvbnji01/nix-2.4pre19700101_eab14a6-x86_64-darwin.tar.xz'; expected @binaryTarball_x86_64-darwin@, got eb40f032d84e2fadb1e875ad7a35a075f9c5b47a4737084a87f925292378b57a

Was anything changed on the installer? Did it already get released and @abathur's installer is deprecated?

Linked issue: digitallyinduced/ihp#960

[domenkozar]

Most likely a transient network error, the hash is correct when I test it locally.

[mpscholten]

Thanks for the quick help Domen! :)

The user reported back that the problem still occurs. Notable it's happening on a Macbook Air with a M1. Maybe this could be causing the problems?

[domenkozar]

The installer that's used by IHP doesn't support M1, I'd suggest to use the newest one from Nix master or use the default installer that ships M1 support as of yesterday.

[mpscholten]

Thanks, using the default installer solved the problem :)

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/what-is-the-recommended-installer-to-use-macos/15138/1