Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

install: configure and bootstrap synthetic.conf on darwin #3212

Merged
merged 12 commits into from May 23, 2020

Conversation

@LnL7
Copy link
Member

LnL7 commented Nov 8, 2019

Fixes #2925.

Here's some preliminary documentation for the apfs volume creation section.

Updated to create an apfs volume instead of using the symlink approach. This works on a clean install but I expect there are other corner cases that are not covered. More testing is probably required.

Using a symlink instead of a separate volume is the most straightforward approach which relies less on darwin specific tools. However it does have some disadvantages, especially if this has to be changed in the future. Also the new location should be avoided for official infrastructure (at least for a while) so installation using an apfs volume should probably also be implemented somewhere.

From my testing it looks like apfs.util works properly now, unless this breaks again a reboot can be bypassed for both mountpoints and symlinks.

@LnL7 LnL7 marked this pull request as ready for review Nov 9, 2019
@qolii
Copy link

qolii commented Nov 9, 2019

Wait, this is viable? From following #2925, I thought the consensus was that nixpkgs currently has impurities in it that preclude using a (symlink to a) non-/nix store path. Did I misunderstand, or is the worry just overblown and this will work for "most people"?

@LnL7
Copy link
Member Author

LnL7 commented Nov 10, 2019

There's a big difference between changing NIX_STORE_DIR and it's physical location. With a symlinked store it's possible for build results to leak the canonical location. While this isn't great, it's is only causes problems when sharing caches between hosts that use different locations. So if all macs use the same target there's no issue. It's a tradeoff, but I feel like it's better to keep the installer as simple as possible.

If we avoid this on the hydra infrastructure that build the official caches (at least for now) I think it won't be a problem anybody runs into in practice.

@lilyball
Copy link
Member

lilyball commented Nov 10, 2019

I am a bit concerned that what I download from Hydra wouldn't match what I build locally, e.g. someone already said llvm (or clang?) seems to do a realpath on the build location and will embed the symlink destination.

As long as apfs.util works, can we not use it for the volume approach?

Another potential benefit of the volume approach that's been mentioned is the fact that we can make it case-sensitive.

@qolii
Copy link

qolii commented Nov 10, 2019

While this isn't great, it's is only causes problems when sharing caches between hosts that use different locations. So if all macs use the same target there's no issue.

Ahh, gotcha. I think I understand now. And as @lilyball says, this would mean that packages built with a "real" store path of /nix might be different to ones we build on a Mac with the symlinked store. But your point is, while they might not match, everything will still work?

@LnL7
Copy link
Member Author

LnL7 commented Nov 10, 2019

Long term this wouldn't be a problem. I'm more concerned about the complexity of managing a separate volume and what could change/break there in the future.

@domenkozar
Copy link
Member

domenkozar commented Nov 10, 2019

cc @edolstra I tested this and it works, it would be great to have 2.3.2 release out once it's reviewed and merged.

@dhess
Copy link

dhess commented Nov 10, 2019

So worst case, if the official macOS builders stay on pre-10.15 for awhile and keep building packages for a physical /nix store location, it just means that Macs that symlink their physical store location to /nix will miss the Hydra cache occasionally, correct?

In any case, I think this should be /opt/nix, because it follows the Linux FHS and therefore there's a better chance (though, still, admittedly slim) that Nix on Linux could adopt it someday, as well.

@lilyball
Copy link
Member

lilyball commented Nov 11, 2019

Are the Hydra macOS builders guaranteed to run on the OS that we use as the minimum macOS target? If we ever run a 10.15 Hydra macOS builder while we still support pre-10.15, that means binary caches for packages that "see through" the symlink will then break on the older OSes.

@mroi
Copy link

mroi commented Nov 11, 2019

I like this solution a lot. However, since I experimented with a symlinked store before, we may need to extend the purity checks in the ld wrapper. I had problems, where the absolute path of the store would sneak into a build process. ld would then complain, that a library from outside the store was linked against, while in fact it was located within the store.

Unfortunately, I do not remember what I was building back then. So this is more of a warning than a call for action right now.

@arcnmx
Copy link
Member

arcnmx commented Nov 13, 2019

Aside from the obvious problems regarding purity, it's worth noting here that I've so far encountered a few inconsistencies in nix from trying to switch to a symlink'd store path:

  • builtins.storePath appears to just not work at all?
  • nix-build /nix/store/something.drv gets confused
    • (need to check if nix-store -r has the same problem?)
@LnL7 LnL7 force-pushed the LnL7:darwin-10.15-install branch from 256b22e to c39c829 Nov 13, 2019
@LnL7
Copy link
Member Author

LnL7 commented Nov 13, 2019

@mroi Things like that I what I was most afraid of, would be great if you had an example.

scripts/install-nix-from-closure.sh Outdated Show resolved Hide resolved
scripts/install-nix-from-closure.sh Outdated Show resolved Hide resolved
scripts/install-nix-from-closure.sh Outdated Show resolved Hide resolved
scripts/install-nix-from-closure.sh Outdated Show resolved Hide resolved
@mroi
Copy link

mroi commented Nov 13, 2019

@mroi Things like that I what I was most afraid of, would be great if you had an example.

I honestly do not remember. Sorry. If I find the time, I will try some things I usually build myself.

@domenkozar
Copy link
Member

domenkozar commented Nov 18, 2019

Note that we'll also need to fix #2407 in order to support macos + multi-user mode.

EDIT: already done in d4e51aa#diff-3f9ecc4aa9722ed5597a741e4cb91d3e

@LnL7 LnL7 force-pushed the LnL7:darwin-10.15-install branch from c39c829 to 7543121 Dec 15, 2019
@@ -176,11 +180,15 @@ let
--absolute-names \
--hard-dereference \
--transform "s,$TMPDIR/install,$dir/install," \
--transform "s,$TMPDIR/create-darwin-volume.sh,$dir/create-darwin-volume.sh," \

This comment has been minimized.

Copy link
@LnL7

LnL7 Dec 15, 2019

Author Member

Not sure why only this file ends up in the wrong place.

@erlend-sh erlend-sh mentioned this pull request Dec 16, 2019
@matthewbauer
Copy link
Member

matthewbauer commented Dec 16, 2019

Looks good besides a few small things. I think it would be good to get this in a Nix release pretty soon as we lose lots of potential users without a well-documented Catalina installer.

Tried on a fresh macOS using the following steps:

# on a pre-10.15 Nix install
$ git clone https://github.com/LnL7/nix
$ cd nix
$ git checkout darwin-10.15-install
$ nix build -f release.nix binaryTarball
$ cp result/*.tar.xz ~/nix.tar.xz
# Copy ~/nix.tar.xz to your Catalina macOS and open a terminal there
$ tar xf nix.tar.xz
$ ./nix-*/create-volume.sh
$ ./nix-*/install.sh --daemon
scripts/create-darwin-volume.sh Outdated Show resolved Hide resolved
scripts/create-darwin-volume.sh Outdated Show resolved Hide resolved
scripts/create-darwin-volume.sh Outdated Show resolved Hide resolved
break
;;
esac
i=$((i+1))

This comment has been minimized.

Copy link
@lilyball

lilyball Dec 16, 2019

Member

This can be let i+=1

}

apfs_volumes_for() {
disk=$1

This comment has been minimized.

Copy link
@lilyball

lilyball Dec 16, 2019

Member

Perhaps local

break
fi
case "$name" in
[Nn]ix*)

This comment has been minimized.

Copy link
@lilyball

lilyball Dec 16, 2019

Member

Since we're using bash this could be written as

if [[ "$name" == [Nn]ix* ]]; then
    echo "$name"
    break
fi
echo " 1. Remove the entry from fstab using 'sudo vifs'"
echo " 2. Destroy the data volume using 'diskutil apfs deleteVolume'"
echo " 3. Delete /etc/synthetic.conf"
echo ""

This comment has been minimized.

Copy link
@lilyball

lilyball Dec 16, 2019

Member

This whole thing could be done as a heredoc

cat >&2 <<EOF

     ------------------------------------------------------------------ 
    | This installer will create a volume for the nix store and        |
    | configure it to mount at /nix.  Follow these steps to uninstall. |
     ------------------------------------------------------------------ 

  1. Remove the entry from fstab using 'sudo vifs'
  2. Destroy the data volume using 'diskutil apfs deleteVolume'
  3. Delete /etc/synthetic.conf

EOF
scripts/create-darwin-volume.sh Outdated Show resolved Hide resolved
scripts/create-darwin-volume.sh Outdated Show resolved Hide resolved
scripts/create-darwin-volume.sh Outdated Show resolved Hide resolved
@LnL7
Copy link
Member Author

LnL7 commented Dec 17, 2019

@lilyball Given that bash is ancient on macOS I intentionally kept this POSIX sh.

@LnL7 LnL7 force-pushed the LnL7:darwin-10.15-install branch from 3832259 to 97c6595 Dec 17, 2019
jopecko added a commit to jopecko/jopecko.github.io that referenced this pull request May 15, 2020
LnL7 and others added 10 commits Dec 15, 2019
Starting macOS 10.15 /nix can't be creasted directly anymore due to the
readonly filesystem, but synthetic.conf was introduced to enable
creating mountpoints or symlinks for special usecases like package
managers.
The default login shell for users on macOS 10.15 changed from bash to
zsh.  So while generally nonstandard we need to configure it to make nix
function out of the box on macOS.
This should handle installation scenarios we can handle with
anything resembling confidence. Goal is approximating the existing
setup--not enforcing a best-practice...

Approaches (+ installer-handled, - manual) and configs each covers:

+ no change needed; /nix OK on boot volume:
  All pre-Catalina (regardless of T2 or FileVault use)

+ create new unencrypted volume:
  Catalina, pre-T2, no FileVault

+ create new encrypted-at-rest volume:
  Catalina, pre-T2, FileVault
  Catalina, T2, no FileVault

- require user to pre-create encrypted volume
  Catalina, T2, FileVault
@LnL7 LnL7 force-pushed the LnL7:darwin-10.15-install branch from f44919c to 2b0a81d May 21, 2020
@LnL7
Copy link
Member Author

LnL7 commented May 21, 2020

The manual and error messages should be much nicer now, uploaded the updated manual here. Since I rebased to fix some merge conflicts it should be tested again but other then that I think it's done.

Thanks @abathur!

@abathur
Copy link
Contributor

abathur commented May 21, 2020

The manual and error messages should be much nicer now, uploaded the updated manual here. Since I rebased to fix some merge conflicts it should be tested again but other then that I think it's done.

Thanks @abathur!

  • @LnL7 tested the installer on Catalina + pre-T2 + FileVault and it correctly prompted to pre-create the volume.
  • I tested Catalina + T2 + (without FileVault & with FileVault). In both cases:
    • a bare call to the installer correctly pointed me to the documentation and prompted me to re-run with the --darwin-use-unencrypted-nix-store-volume flag
    • re-running with the flag correctly created the volume and installed Nix
  • I also tested Mojave + pre-T2 + (without FileVault & with FileVault, for good measure) and in both cases it cleanly installed on a bare call to the installer.
@domenkozar
Copy link
Member

domenkozar commented May 23, 2020

I'm going to merge this, given that people have been reporting for months that it works (and now that documentation has been written) it seems like not much will improve until we get this live.

@domenkozar domenkozar merged commit c129e7c into NixOS:master May 23, 2020
2 checks passed
2 checks passed
tests (ubuntu-latest)
Details
tests (macos-latest)
Details
@ProofOfKeags
Copy link

ProofOfKeags commented May 24, 2020

So either I'm missing something or the instructions in the manual definitely don't work. It complains about asking for the flag about --daemon | --no-daemon and I cannot seem to supply either argument. It isn't threading the flags through to the installer correctly. What's going on here?

@abathur
Copy link
Contributor

abathur commented May 24, 2020

@ProofOfKeags How are you building/obtaining the install script? What exact command are you using?

@ProofOfKeags
Copy link

ProofOfKeags commented May 24, 2020

$ sh <(curl https://nixos.org/nix/install) --darwin-use-unencrypted-nix-store-volume as stated in the manual link

EDIT: after digging it seems apparent that whatever install script I'm fetching does not even include the option for the --darwin... I gather this hasn't been pushed out to the main host?

@abathur
Copy link
Contributor

abathur commented May 24, 2020

A new version of Nix hasn't been released yet.

Edit: I don't know exactly what the release/deploy cycle for Nix looks like, but (AFAIK) if you watch the repo for a new release, the command you quoted should work at approximately that time :)

@abathur
Copy link
Contributor

abathur commented May 25, 2020

Until it's released, my macOS system bootstrap has been doing roughly (I'm adapting it for you--I last ran it before any of this work was merged...) this:

curl https://raw.githubusercontent.com/NixOS/nix/master/scripts/create-darwin-volume.sh | bash
curl https://nixos.org/nix/install | sh

(create-darwin-volume.sh is what the --darwin-use-unencrypted-nix-store-volume flag will cause the main installer to run; it needs to be run before the main install)

@abathur
Copy link
Contributor

abathur commented May 25, 2020

@ProofOfKeags Aside from this mishap: You may be one of the first to try following the manual update from scratch--any feedback? Do you have questions it didn't answer?

@ProofOfKeags
Copy link

ProofOfKeags commented May 25, 2020

The guide seemed fairly straight forward, I think. But given that the first command didn't work and I didn't bother trying to go through the rest since it'll work when it is released, I can't say for sure whether the instructions for the rest of it will work. I actually have half a mind right now to partition my mac again and just use NixOS raw, because MacOS seems to be increasingly hostile towards developers.

@nathanshelly
Copy link

nathanshelly commented May 25, 2020

@abathur just for confirmation I was successfully able to remove my single user-install and reinstall a multi-user installation on Catalina + T2 + Filevault earlier today running those same commands

@sengaya
Copy link

sengaya commented May 29, 2020

@abathur I can confirm that I could successfully set up nix on Catalina (with T2 chip) with the commands

curl https://raw.githubusercontent.com/NixOS/nix/master/scripts/create-darwin-volume.sh | bash
curl https://nixos.org/nix/install | sh
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

You can’t perform that action at this time.