/nix will not be writable on macOS Catalina

[mroi]

This is not a short term bug, but it will become an issue when macOS Catalina is released this fall. macOS is now split across two volumes (system and data) with a read-only system volume. This means that /nix will no longer be writable.

Some more information can be found in the related WWDC talk and some session notes people took from a Q&A.

Summary: the system volume, which is mounted at / will become non-writable. Some directories that need to be writable are connected via firmlinks (an Apple invention) to the data volume. /nix is not among these locations, so with the release of macOS Catalina, this location is no longer an option.

I see two possible solutions:

1. I could try and file a bug to convince Apple to pre-install /nix as a firmlink to a writable location. I think this has limited success and Nix would then depend on Apple to not drop this link in a future release.
2. We could move Nix on macOS to a different default location. Possible locations are those that Apple chooses to pre-install as links to writable locations. Two examples are /usr/local and /opt, so we could move to /usr/local/nix or /opt/nix. I would hope that these locations are common enough so that Apple would not drop them in the future.

I wanted to raise this issue early, before it becomes a problem for users. If this issue tracker is not the right place, please feel free to move this discussion elsewhere. I would also be available for testing any potential solution, since I have access to a macOS Catalina beta.

[edolstra]

Realistically, this may force us to drop support for macOS. Using a different store location for macOS would require a separate binary cache and a separate Hydra instance, and there would be no guarantee that the new location wouldn't break in the future.

I did notice in the talk that it's still possible to make the system volume writable, though not in a persistent way.

[mroi]

Speaking for myself: I would be quite sad if macOS support would go away.
I will file a bug with Apple and see what happens. Cannot hurt.

Yes, you can jump through hoops and get a writable system volume, but only until the next reboot, so this is not a useful option.

[jwiegley]

@edolstra Before canceling support, we can ask the community if it's willing to host its own binary cache and Hydra instance, just as we've enlisted a host of volunteers to keep nixpkgs at near parity on Darwin. Having the answer be, "This doesn't easily fit our model so we won't support Mac" would be an unfortunate response.

[mroi]

I think no such decision should be made lightly, but it is definitely a drastic change on Apple’s part.

[edolstra]

Yes, you can jump through hoops and get a writable system volume, but only until the next reboot, so this is not a useful option.

Maybe you can use this to create a /nix symlink to another volume, which would hopefully persist across reboots?

[mroi]

That was my first try to get it temporarily working again. But nix-env complains then:

error: cannot open connection to remote store 'daemon': the path '/nix' is a symlink; this is not allowed for the Nix store and its parent directories

If that error can be safely ignored, I think this would be a workaround worth exploring. The way I understand things, anything you place in the root folder would persist until the next system update.

[LnL7]

There's no real reason for the nix store to be on the system volume, with apfs it's easy to create a separate volume and mount it to /nix. But perhaps there's something I'm missing because I don't understand the need for these firmlinks, the equivalent is possible with mounts similar to how the store is mounted readonly on NixOS.

[edolstra]

@mroi You can set NIX_IGNORE_SYMLINK_STORE=1 to disable that check. (This should probably be turned into a nix.conf option.)

[jwiegley]

@edolstra Great idea, added #2926.

[mroi]

@edolstra Thanks for the tip, this is a useful workaround.

But future Nix-users on macOS still need to find a reliable way to inject this symlink into the read-only root directory. This probably involves some advanced steps like rebooting into recovery mode after every system upgrade.

@LnL7 The same would be true for /nix as a mount point, because we would need to create this directory on the read-only volume.

[LnL7]

Sure but if Apple doesn't want to provide some sort of solution for one of these cases there will be no alternative to changing the default prefix. Which is arguably is not worth it for a secondary platform that's continuously moving further away from UNIX.

Is there an overview of the full volume layout somewhere? /usr/local and /Users are used as examples but I can't imagine that's everything.

[kevingriffin]

@mroi How have you been testing this? In my testing, I was able to install nix, but I assumed it was due to the comment in the presentation that "system volume is writeable in the developer preview". I'd also like to help test, but was able to get nix installed:

kevingriffin@KevinnoiMac /nix % uname -a
Darwin KevinnoiMac.local 19.0.0 Darwin Kernel Version 19.0.0: Fri May 24 17:36:10 PDT 2019; root:xnu-6041.0.0.111.5~1/RELEASE_X86_64 x86_64
kevingriffin@KevinnoiMac /nix % ls /nix
store	var
kevingriffin@KevinnoiMac /nix % nix-env -iA nixpkgs.hello
installing 'hello-2.10'
these paths will be fetched (0.02 MiB download, 0.07 MiB unpacked):
  /nix/store/c4w9z1kzzkdsgvgr6cy9ggl39s2yzn70-hello-2.10
copying path '/nix/store/c4w9z1kzzkdsgvgr6cy9ggl39s2yzn70-hello-2.10' from 'https://cache.nixos.org'...
building '/nix/store/iwcsfsh5hxyzpymnkp1r54nb3zzpd7mg-user-environment.drv'...
created 2 symlinks in user environment```

[matthewbauer]

It looks like you can still access the /nix directory after the update, it's just in /System/Volumes/Data/nix. Too bad there's no chroot equivalent on macOS.

[matthewbauer]

sudo mount -uw / && sudo ln -s /System/Volumes/Data/nix /nix works for me! This is on SIP even! Having the installer handle this might work.

[kevingriffin]

I wonder if that will continue to work in the later builds, when they plan to enforce the read-only aspect.

[mroi]

Apple says in the WWDC talk, that the first beta seeds mount the system volume read-write, but this is only temporary. A later seed will turn it read-only. You can create a file /.rootro to opt-in to the read-only mount now.

[mroi]

But there is good news. I talked to an Apple engineer on Twitter and they are working on a solution to create symlinks in the root directory even when the system volume is mounted read-only. I guess we’ll just wait and see how this turns out.

For now, symlinking /nix to /System/Volumes/Data/nix in combination with NIX_IGNORE_SYMLINK_STORE=1 is a viable workaround for everyone on the macOS Catalina testing train.

[angerman]

@grahamc remember our discussion about symlink /nix and the installer during ZuriHac? Turns out, if I had read the issue tracker more closely I had known about NIX_IGNORE_SYMLINK_STORE=1, and wouldn't have been as annoyed.

[matthewbauer]

This appears to be fixed in 10.15 Beta 3! I can create a /nix directory and use it as a store, while /System, /bin, /sbin, etc. are still read only. Users upgrading from previous beta versions can do:

sudo rm -f /nix
sudo mv /System/Volumes/Data/nix /nix

then start the daemon if you are in multi user:

sudo launchctl load /Library/LaunchDaemons/org.nixos.nix-daemon.plist
sudo launchctl start org.nixos.nix-daemon

New installs should work out of the box.

[EHYPERCHICKEN]

/ will ship as non-writeable -- what you're seeing is a known issue where it can become writeable due to another system component. If you reboot, you will not see this work.

[matthewbauer]

/ will ship as non-writeable -- what you're seeing is a known issue where it can become writeable due to another system component. If you reboot, you will not see this work.

Will sudo mount -uw / continue to work?

[EHYPERCHICKEN]

Will sudo mount -uw / continue to work?

If you disable SIP, yes.

[matthewbauer]

Will sudo mount -uw / continue to work?

If you disable SIP, yes.

Ugh, I don't think we want to recommend that. Are any exceptions for Nix possible? I saw that Fink's /sw is included in root now, if we could get something like this for Nix it would be extremely helpful.

To be clear, changing the root directory of Nix from /nix to something else is possible, but also very painful. We would lose some resource sharing abilities between macOS & Linux, as well as require setting up a special binary cache. The only alternative to this is recommending to our users insecure practices like disabling SIP.

[EHYPERCHICKEN]

We're working on something that will allow you to create either a symlink or empty directory at / even though it's read-only. So you could go one of two ways:

1. Create /nix symlink to the data volume
2. Put the repo on a separate apfs volume that space shares within the container, create a /nix empty directory, and then mount that volume on that directory

[matthewbauer]

We're working on something that will allow you to create either a symlink or empty directory at / even though it's read-only. So you could go one of two ways:

1. Create /nix symlink to the data volume
2. Put the repo on a separate apfs volume that space shares within the container, create a /nix empty directory, and then mount that volume on that directory

Sounds reasonable! The main concern with (1) is that realpath(3) will still report /System/Volumes/Data/nix when pwd=/nix (#2926). That can get /System/Volumes/Data/nix hardcoded for some build systems. This could be a problem when that directory exists on Catalina but not on other machines. We need something a little bit stronger than a symlink ;)

[mroi]

If we were to mount Nix from a separate volume, we could use the opportunity to use the case-sensitive version of APFS. This would remove the need to fix weirdo packages relying on files that just differ in case.

[EHYPERCHICKEN]

I never object to a change which spreads the case-sensitivity gospel.

[angerman]

If we were to mount Nix from a separate volume, we could use the opportunity to use the case-sensitive version of APFS. This would remove the need to fix weirdo packages relying on files that just differ in case.

I have been doing this with great success (although with hfs+ case sensitive) with an external ssd for a while now. The primary motivation being, that I can take my nix store from my desktop with me when I use my laptop while walking about.

[lunaticare]

I use this script to create volumes in semi-automated fashion with FileVault enabled:

export CRYPTO_USER=$(sudo dscl . read /Users/$(whoami) | grep 'GeneratedUID' | cut -d ' ' -f2)
export PASSPHRASE=$(pwgen -C 20)

/nix volume:

sudo diskutil apfs addVolume disk1 APFSX Nix -mountpoint /nix
sudo diskutil enableOwnership /nix
echo $PASSPHRASE | sudo diskutil apfs encrypt /nix -user disk -stdinpassphrase
echo $PASSPHRASE | sudo diskutil apfs encrypt /nix -user $CRYPTO_USER  -stdinpassphrase
sudo chflags hidden /nix
echo 'LABEL=Nix /nix apfs rw' tee -a /etc/fstab
UUID=$(diskutil info -plist /nix | plutil -extract VolumeUUID xml1 - -o - | plutil -p - | sed -e 's/"//g')
security add-generic-password -l Nix -a "$UUID" -s "$UUID" -D "Encrypted Volume Password" -w "$PASSPHRASE" \
 -T "/System/Library/CoreServices/APFSUserAgent" -T "/System/Library/CoreServices/CSUserAgent"

Same for /run:

sudo diskutil apfs addVolume disk1 APFSX Run -mountpoint /run
sudo diskutil enableOwnership /run
echo $PASSPHRASE | sudo diskutil apfs encrypt /run -user disk -stdinpassphrase
echo $PASSPHRASE | sudo diskutil apfs encrypt /run -user $CRYPTO_USER -stdinpassphrase
sudo chflags hidden /run
# sudo chown -R <username> /run
echo 'LABEL=Run /run apfs rw' | sudo tee -a /etc/fstab
UUID=$(diskutil info -plist /run | plutil -extract VolumeUUID xml1 - -o - | plutil -p - | sed -e 's/"//g')
security add-generic-password -l Run -a "$UUID" -s "$UUID" -D "Encrypted Volume Password" -w "$PASSPHRASE" \
 -T "/System/Library/CoreServices/APFSUserAgent" -T "/System/Library/CoreServices/CSUserAgent"

[johanatan]

This is pretty complicated just for trying to install a package manager. Any ETA on when the installer instructions themselves will be "fixed"? I don't necessarily want to dive into all of these details and would prefer something that "just works".

[druimalban]

It may be a bit complicated but it is not exactly helped by people continuing to post the same things over and over again. Some comment (now swallowed up, never to be seen again) foreshadowed exactly this and warned against it

[johanatan]

Yes, someone predicted this was going to be a problem with the OS update that was released months ago. They even did so during the "beta" phase which offered plenty of time to get ahead of the problem and fix the installation at the main landing page where people from all over the world are continually landing to this day. :)

[druimalban]

No, it was this comment: #2925 (comment)

and it was right

[johanatan]

I was thinking of a different (and earlier one) obviously. Perhaps on a completely different issue that I went through before landing here.

[johanatan]

And that's still ridiculously complicated. I would say that your software project is failing if it requires reading a wall of text and issuing lots of volume-based commands just to get a basic installation complete. Major fail in fact.

[lunaticare]

@druimalban @johanatan will it be sane to offer interactive script to setup raw/encrypted volumes for a single user installation?
What a good solution might look like?

[alper]

Unsubscribing here for the time being. Somebody let me know when this is usable for a casual macOS user because this does not seem like it.

[abathur]

This limbo is frustrating from every angle, but we won't get anywhere good by venting at people (even, I suppose, at Apple?)

Having enjoyed using Nix on macOS over the past 2+ years, it's frustrating to see this keep away prospective users. That said, it's also abnormal for a "basic installation" to have to go create new volumes (let alone encrypted ones) and I think we'd be in an even worse place if the installer was out in the wild doing it in a half-baked way.

[johanatan]

This installer is out in the wild doing things in a half-baked way. I have no idea the state of my system in fact nor do I have any way to completely erase whatever Nix did to it. /nix remains undeletable and God knows what else Nix did before it crashed.

[abathur]

Did you follow the removal steps you quoted in #2925 (comment)?

[johanatan]

Not yet. That's the first I've heard of any "removal steps". I didn't read that comment in detail yet--just saw a wall of text and said "that'll have to wait until later".

[samuela]

What's the status on this issue? The website and installer are quite misleading in the sense that they still claim complete compatibility with macOS. Is there a path forward to using Nix on macOS without any hacks?

[wmertens]

@samuela macOS does not allow simply creating /nix. So that needs to be configured in some way, not really a "hack".

The easiest way to have Nix on macOS would be if Nix used /opt/nix or /usr/local/nix instead of /nix.

[samuela]

So what is the problem with /opt/nix or /usr/local/nix/ instead of /nix? This seems like a real showstopper bug...

[wmertens]

@samuela it means either doubling the storage for Hydra, or changing ALL the Nix installs everywhere.

One hacky option is to rewrite store paths as they get downloaded. So any mention of /nix/store/...name would be rewritten in the file to /opt/nix/...(name-4chars). This would work for most of the files, but still needs support in nix-store and all over nixpkgs where /nix is hardcoded

[alper]

@wmertens How much storage would get doubled?

[wmertens]

@alper any package that Hydra produces would have to be produced twice, once for /nix and once for /opt/nix.

It's not quite a doubling, because not all packages are built for Darwin and files that don't reference store paths would likely be the same and can be hardlinked.

I don't know how big the Hydra storage is.

[samuela]

Seeing as this issue has been open since 2019, it seems as though macOS is no longer a supported platform. Is that true? Or is there a roadmap to fixing this?

[alper]

Because shouldn’t storage be a non-issue at this point in history?

[grahamc]

I'm going to lock this until we're able to get a fix out. Thanks for everybody's contributions and help in figuring out this problem, hopefully it is soon.

[edolstra]

@wmertens We wouldn't need to produce packages twice, since we'd have one Hydra instance (and binary cache) for macOS (using the /opt prefix) and another instance for all other platforms.

The main consequence of using a separate prefix for macOS is that you can't have Hydra jobsets anymore containing jobs for macOS and Linux. It would also make it harder to deploy from macOS to Linux.

[domenkozar]

Documentation for installation of Nix on Catalina is available.

Note that until Nix 2.3.5 release is out, the instructions won't work. PR

[domenkozar]

Nix 2.3.5 has been released with Catalina fix!