Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

darwin: encrypt nix volume if filevault is enabled #4289

Merged
merged 1 commit into from
Apr 29, 2021

Conversation

abathur
Copy link
Member

@abathur abathur commented Nov 28, 2020

This refactors WIP from #4181 to address things that became obvious:

  1. The macOS install is getting too complex for single-user mode.
  2. There's already frustration around difficulty of uninstalling multi-user on macOS.
  3. I want to support upgrading existing macOS volumes to work the same as those made by this WIP, but simply tacking it on as an extra option at all relevant points was looking like a lot of extra conditional logic and calls to some fairly slow diskutils commands.
  4. The greater number of Nix-related artifacts shook loose additional concern around avoiding state/artifact inconsistencies.
  5. As I developed a curing process to address these inconsistencies, I realized it won't take too much more work to extend these concepts to uninstalling the other Darwin and general Nix multi-user artifacts. I'm pleasantly surprised how practical an uninstall mode feels from here.

The history had grown unruly even before this refactor, so for now I'm flattening this into an omnibus commit to start fresh. Since this touches on a lot, I've tried to better-document the high-altitude view by making three outlines:

  • the general process/logic in pseudocode (as-is, and for the uninstaller I'm imagining) removed this for merge
  • how high-level user-facing functionality changes here
  • what lower-level implementation changes I'm making

High-level functionality changes

  • disable single-user installs on macOS
  • if the user's root is read-only, create a volume (overridable default)
    • --darwin-use-unencrypted-nix-store-volume flag is now a no-op warning
    • if the boot volume has FileVault enabled, encrypt this volume
      • generate a random credential and place it in the system keychain
    • create a LaunchDaemon to mount it
      • still use fstab to specify mount opts (rw,noauto,nobrowse,suid,owners)
        • noauto so that the system won't auto-mount if our mounter breaks (goal: the user finds/presents for help)
      • the daemon isn't strictly necessary for unencrypted volumes, but I'm using it in both cases for consistency
  • update volume naming/customization
    • default: use or create volume named "Nix Store" on same disk as /
    • overridable variables for the filesystem, label, and target disk
    • overridable variables to BYOV: volume special device, and UUID
  • support curing any volume-related state (if present) on install
    • if Nix Store is encrypted
      • verify we can unlock it from keychain
      • otherwise, prompt the user to delete it or manually add the pw to keychain
    • if filevault is on but Nix Store isn't encrypted, offer to encrypt it
    • if nix is the only item in fstab and synthetic.conf, remove them; otherwise suggest edits that will remove the nix line
    • remove the mounter launchdaemon

Lower-level implementation changes

  • scaffold uninstall and combined uninstall+install actions
  • multi-user installer
    • add reminder/remind UI/X idiom (add notes as we go, show all at end)
      • no more saving variables or re-testing conditions at the end
      • if interactive: show one at a time and require acknowledgement
      • convert some existing messages to use this
    • add password_confirm UI/X idiom for actions like deleting a volume
      • forces a sudo prompt (even when headless or sudo was recently authed)
    • add confirm_rm and confirm_edit UI/X idioms
      • these are "for" curing volume state, but I anticipate them being useful for uninstaller (and, I guess, for the installer if we wanted to be more interactive/explicit around some edits)
    • try to make create_directories more idempotent (may cause trouble?)
      • if /nix already exists, chown -R root:nixbld it
      • use install -dv -m instead of mkdir -pv -m
      • combine dir-structure steps 2 & 3 with install's -g (group) flag
    • only defer to a pre-set NIX_SSL_CERT_FILE if its physical path is not in /nix/store
      • multi-user reinstall over a single-user install was skipping cert setup
    • fail earlier (before sudo lecture etc.) if EUID=0
    • add two new install phases:
      • (poly_)cure_artifacts for reverting changes Nix makes
      • poly_prepare_to_install for preparing the Nix volume on macOS
    • normalize SCRATCH/mktemp path when macOS /usr/bin/mktemp is used:
      • /usr/bin/mktemp -d -t tmp.XXXXXXXXXX --> /.../tmp.XXXXXXXXXX.2RMrVwkx
      • /usr/bin/mktemp -d "${TMPDIR:-/tmp/}tmp.XXXXXXXXXX" --> /.../tmp.JwmZDxN1kf
    • _sudo "explanation" command ... wrapper now sends explanation to stderr
      • make it easier to pipe commands using the wrapper; see create-darwin-volume.sh:volume_pass_works() for an example
  • refactor create-darwin-volume.sh to
    • be sourceable as a library but still support direct invocation
      • use UI/X idioms from the multi-user installer
      • give multi-user installer fine-grained control of what runs when
      • ~legacy (bare) direct invocation
        • in case we have to reinstate single-user
        • avoid no/low-notice breaks for anyone who has been directly running it
      • externally invoke a non-main func with args "no-main" "funcname" ...
      • I've seen a fair number of modified forks of this script; over time I hope sourceability and external function invocation will make it easier to re-use
    • largely eradicate a pattern, <run very slow diskutil command> | <parse a single bit of info out with an xpath query>, which is OK in isolation but was becoming a bit too common
    • remove (most) instruction dumps that describe how to remove things that the curing process can now handle
    • defensively use absolute default paths for some common executables (coreutils, find, grep, etc.)

@LnL7 @lilyball @matthewbauer @dhess @mroi @domenkozar (others I'm forgetting are inevitably also interested; feel free to add people)

Try it out

I have been hosting my own installers for much of the history of this PR. Nix now has a better installer-testing process, so I'm going to see if we can lean on it here. I'll leave the old process note here for a bit in case there are problems with the new approach. If you can't try it out, you can see output from the CI run.

The new approach requires an extra flag, so it'll be easiest to just use in interactive mode:

sh <(curl https://abathur-nix-install-tests.cachix.org/serve/yihf8zbs0jwph2rs9qfh80dnilijxdi2/install) --tarball-url-prefix https://abathur-nix-install-tests.cachix.org/serve

Here's a history of the versions it has pointed at, with the current at the bottom (note: I'll only update this for substantive changes or breakage):

old privately-hosted installer (out of date--use only if above doesn't work)

I have a temporary installer up at https://files.t-ravis.com/install-multi, so curl https://files.t-ravis.com/install-multi | sh (headless) and sh <(curl https://files.t-ravis.com/install-multi) (interactive) should work.

Other issues

I believe this PR:

@dhess
Copy link

dhess commented Nov 28, 2020

Thanks once again for this work and the attention to detail.

Does anyone know: on T2 Macs, if you encrypt a user-created APFS volume, is the encryption/decryption for that volume still done on the T2, as with the system volumes; or is it done in software on the CPU?

@abathur
Copy link
Member Author

abathur commented Nov 28, 2020

on T2 Macs, if you encrypt a user-created APFS volume, is the encryption/decryption for that volume still done on the T2, as with the system volumes; or is it done in software on the CPU?

I think it's on the T2, but I won't swear I personally know this as a fact. For example, see the last line below:

$ diskutil apfs encryptVolume disk1s7 -user disk -passphrase foo
Encrypting with the new "Disk" crypto user on disk1s7
The new "Disk" user will be the only one who has initial access to disk1s7
The new APFS crypto user UUID will be 920D4EAD-BFAE-457B-9A19-827AAC0C1B42
Encryption has likely completed due to AES hardware; see "diskutil apfs list"

If you encrypt a volume on a pre-T2 mac, that line will say Background encryption is ongoing; see "diskutil apfs list" to see progress

@LnL7
Copy link
Member

LnL7 commented Nov 28, 2020

That's also how I understood the T2 documentation from apple https://www.apple.com/jp/mac/docs/Apple_T2_Security_Chip_Overview.pdf.

The chip handles all the encryption and is actually always enabled, FileVault only changes the conditions for which the chip unlocks the data. Meaning that the data also doesn't have to change when updating credentials. Which the behaviour you noticed seems to confirm is the case.

@abathur
Copy link
Member Author

abathur commented Nov 28, 2020

In case anyone's curious, I added screenshots of the ~interactive UI/X elements in a comment alongside the code for each (and immediately resolved them, so they don't take up gobs of space in the main thread).

@lilyball
Copy link
Member

lilyball commented Dec 1, 2020

For future reference, you can use <details> tags to hide screenshots by default.

<details><summary>Description of screenshot</summary>

// drag image into this blank line, if it's surrounded by blank lines the markdown ![image](url) syntax will work just fine.

</details>

This will show up as

Collapsed summary here

Once expanded the image will be visible.

Copy link
Member

@lilyball lilyball left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is only a partial review, I’ll have to come back to this later.

doc/manual/src/installation/installing-binary.md Outdated Show resolved Hide resolved
scripts/create-darwin-volume.sh Outdated Show resolved Hide resolved
scripts/create-darwin-volume.sh Outdated Show resolved Hide resolved
scripts/create-darwin-volume.sh Outdated Show resolved Hide resolved
scripts/create-darwin-volume.sh Outdated Show resolved Hide resolved
scripts/create-darwin-volume.sh Outdated Show resolved Hide resolved
scripts/create-darwin-volume.sh Outdated Show resolved Hide resolved
scripts/create-darwin-volume.sh Outdated Show resolved Hide resolved
scripts/create-darwin-volume.sh Outdated Show resolved Hide resolved
scripts/create-darwin-volume.sh Outdated Show resolved Hide resolved
@abathur abathur marked this pull request as ready for review December 6, 2020 03:34
@abathur
Copy link
Member Author

abathur commented Dec 6, 2020

I've pushed commits addressing one round of comments from @lilyball and @SuperSandro2000 (also thanks to @samueldr and @jtojnar for additional discussion). Going to go ahead and convert this from a draft.

I've started a build of this locally, and will update to confirm its status (status: my install checks are working as of 6bb3bc9, but the Nix repo checks failed at this commit--I'd be surprised if this caused that, so I assume it's a transient failure or flakiness?)

For anyone champing at the bit, I suspect we'll be holding up just a little bit (if it's a short wait) for the chance I'll be able to help @domenkozar test a new GitHub Actions integration per #4047 that should make it easier to test PRs that touch the installer.

In any case, reviews still welcome in the meantime. :)

Copy link
Member

@lilyball lilyball left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another partial review. Apologies for not managing to do a full one.

scripts/create-darwin-volume.sh Outdated Show resolved Hide resolved
@@ -1,33 +1,235 @@
#!/bin/sh
set -e
#!/usr/bin/env bash
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure if this really matters, but maybe we should stick with /bin/bash just to ensure we're using the exact same version of bash everywhere?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good question.

I just adopted the shebang from install-multi-user.sh for consistency.

I'm not sure how to weight consistency with install-multi-user against being explicit here. Everything install-multi-user.sh sources has to be 3.2 compatible for this reason (and I did a lot of manual 3.2 testing across all of the scripts as I went).

I assumed/inferred from the #!/usr/bin/env bash shebang on install-multi-user that at least some of the platforms we might be run on don't have a /bin/bash, but if that's not true I suppose they could all change.

What do you think?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suppose consistency is worthwhile. If we do need to run install-multi-user.sh on a platform that doesn't have /bin/bash then the current shebang makes sense.

scripts/create-darwin-volume.sh Outdated Show resolved Hide resolved
scripts/create-darwin-volume.sh Outdated Show resolved Hide resolved
scripts/create-darwin-volume.sh Outdated Show resolved Hide resolved
scripts/create-darwin-volume.sh Outdated Show resolved Hide resolved
scripts/create-darwin-volume.sh Outdated Show resolved Hide resolved
scripts/create-darwin-volume.sh Outdated Show resolved Hide resolved
scripts/create-darwin-volume.sh Outdated Show resolved Hide resolved
scripts/create-darwin-volume.sh Outdated Show resolved Hide resolved
@abathur
Copy link
Member Author

abathur commented Dec 20, 2020

bf5290d fixes an oversight caught by lilyball regarding curing/checking for the presence of Nix as a symlink in synthetic.conf. You can see sample output in this CI run:

The stitched jobs fail because once we've stitched, we need a reboot to remove the symlink.

@ethnt
Copy link

ethnt commented Jan 2, 2021

@abathur Thanks a lot for your work on this! Is there an ETA for when this will be made publicly available?

@abathur
Copy link
Member Author

abathur commented Jan 2, 2021

Not sure. We're on the crowded side of a bottleneck. :)

@jgeerds
Copy link
Member

jgeerds commented Apr 30, 2021

@abathur Thank you very much for your work! Am I right that these changes only affect nixUnstable and upcoming releases? If so, is there a chance for a 2.3.x backport?

@abathur
Copy link
Member Author

abathur commented Apr 30, 2021

@jgeerds Yep.

Backport decisions are above my paygrade, but unless 2.4 is impending I think it spares enough pain to be worth a backport and release.

@abathur
Copy link
Member Author

abathur commented May 1, 2021

This may also be a good place to note: I think the merge of this PR lights a path to full uninstall/reinstall. I can't pursue it any time soon, but I'm happy to discuss as needed if someone else can. I wrote a proof-of-concept in Oct. that should be reheatable. Validation is also much easier since domen added installer tests.

@nixos-discourse
Copy link

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nix-macos-monthly/12330/2

@chkoss chkoss mentioned this pull request May 6, 2021
@rpearce
Copy link

rpearce commented May 13, 2021

Holy smokes; what a great install experience. This worked for me on a macOS early 2013 running 10.15.7. Thank you!!!

@rpearce
Copy link

rpearce commented May 18, 2021

@abathur Sorry to bother: how will we know when this will become the default installer for Darwin? This is what I'm telling coworkers currently:

sh <(curl https://abathur-nix-install-tests.cachix.org/serve/yihf8zbs0jwph2rs9qfh80dnilijxdi2/install) --tarball-url-prefix https://abathur-nix-install-tests.cachix.org/serve

@abathur
Copy link
Member Author

abathur commented May 18, 2021

@rpearce No worries. Subscribing to this thread is probably the best bet.

I deferred a sprawling, intricate refactor of another project for a few months to focus on this PR. For my own sanity, I had to stick a fork in this a couple months back so that I could shift enough of my focus back to that effort to make any headway.

I guess we're waiting for either:

  • a new release from master (whether that's 2.4 or 3.0)
  • someone to drive an effort to backport it to the 2.3-maintenance branch and get a new release from there

It can be backported (and I agree with releasing it), but there are a few caveats that probably make it more complicated than just bugging domen:

  1. There are other commits in master that haven't been backported, and this PR is extensive enough that I imagine someone would need to audit the other changes there to make sure anything this builds on (and any fixes to this?) get pulled in. A good place to start is probably something like: git log --color=always --format='%C(auto,blue)%h%Creset %C(auto,green)%cs%Creset [%C(auto,red)%<|(37)%S%Creset]%C(auto,yellow) %m %Creset%s' --cherry-mark --no-merges master...2.3-maintenance -- scripts | grep -v ' = '
  2. The documentation format was converted from xml to markdown between 2.3-maintenance and this work, so the doc changes will need to be re-applied in the xml.
  3. The installer-testing work domen implemented this winter isn't present in 2.3-maintenance, so someone will need to either backport that work as well, or manually generate installers to test them (i.e., the old way :/)

@nixos-discourse
Copy link

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/anyone-up-for-picking-at-some-nix-onboarding-improvements/13152/1

@nixos-discourse
Copy link

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/how-to-install-nix-on-macos-with-m1/13791/4

@mpscholten
Copy link
Contributor

An issue was just reported to IHP with the following error running this installer:

SHA-256 hash mismatch in 'https://abathur-nix-install-tests.cachix.org/serve/jqr52m6w8dwag5932hw67kq0rvbnji01/nix-2.4pre19700101_eab14a6-x86_64-darwin.tar.xz'; expected @binaryTarball_x86_64-darwin@, got eb40f032d84e2fadb1e875ad7a35a075f9c5b47a4737084a87f925292378b57a

Was anything changed on the installer? Did it already get released and @abathur's installer is deprecated?

Linked issue: digitallyinduced/ihp#960

@domenkozar
Copy link
Member

Most likely a transient network error, the hash is correct when I test it locally.

@mpscholten
Copy link
Contributor

Thanks for the quick help Domen! :)

The user reported back that the problem still occurs. Notable it's happening on a Macbook Air with a M1. Maybe this could be causing the problems?

@domenkozar
Copy link
Member

The installer that's used by IHP doesn't support M1, I'd suggest to use the newest one from Nix master or use the default installer that ships M1 support as of yesterday.

@mpscholten
Copy link
Contributor

Thanks, using the default installer solved the problem :)

@nixos-discourse
Copy link

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/what-is-the-recommended-installer-to-use-macos/15138/1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment