Skip to content
GitHub no longer supports this web browser. Learn more about the browsers we support.
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

/nix will not be writable on macOS Catalina #2925

Open
mroi opened this issue Jun 6, 2019 · 255 comments · May be fixed by #3212
Open

/nix will not be writable on macOS Catalina #2925

mroi opened this issue Jun 6, 2019 · 255 comments · May be fixed by #3212
Labels

Comments

@mroi
Copy link

@mroi mroi commented Jun 6, 2019

This is not a short term bug, but it will become an issue when macOS Catalina is released this fall. macOS is now split across two volumes (system and data) with a read-only system volume. This means that /nix will no longer be writable.

Some more information can be found in the related WWDC talk and some session notes people took from a Q&A.

Summary: the system volume, which is mounted at / will become non-writable. Some directories that need to be writable are connected via firmlinks (an Apple invention) to the data volume. /nix is not among these locations, so with the release of macOS Catalina, this location is no longer an option.

I see two possible solutions:

  1. I could try and file a bug to convince Apple to pre-install /nix as a firmlink to a writable location. I think this has limited success and Nix would then depend on Apple to not drop this link in a future release.
  2. We could move Nix on macOS to a different default location. Possible locations are those that Apple chooses to pre-install as links to writable locations. Two examples are /usr/local and /opt, so we could move to /usr/local/nix or /opt/nix. I would hope that these locations are common enough so that Apple would not drop them in the future.

I wanted to raise this issue early, before it becomes a problem for users. If this issue tracker is not the right place, please feel free to move this discussion elsewhere. I would also be available for testing any potential solution, since I have access to a macOS Catalina beta.

@domenkozar domenkozar added the darwin label Jun 6, 2019
@edolstra

This comment has been minimized.

Copy link
Member

@edolstra edolstra commented Jun 6, 2019

Realistically, this may force us to drop support for macOS. Using a different store location for macOS would require a separate binary cache and a separate Hydra instance, and there would be no guarantee that the new location wouldn't break in the future.

I did notice in the talk that it's still possible to make the system volume writable, though not in a persistent way.

@mroi

This comment has been minimized.

Copy link
Author

@mroi mroi commented Jun 6, 2019

Speaking for myself: I would be quite sad if macOS support would go away.
I will file a bug with Apple and see what happens. Cannot hurt.

Yes, you can jump through hoops and get a writable system volume, but only until the next reboot, so this is not a useful option.

@jwiegley

This comment has been minimized.

Copy link

@jwiegley jwiegley commented Jun 6, 2019

@edolstra Before canceling support, we can ask the community if it's willing to host its own binary cache and Hydra instance, just as we've enlisted a host of volunteers to keep nixpkgs at near parity on Darwin. Having the answer be, "This doesn't easily fit our model so we won't support Mac" would be an unfortunate response.

@mroi

This comment has been minimized.

Copy link
Author

@mroi mroi commented Jun 6, 2019

I think no such decision should be made lightly, but it is definitely a drastic change on Apple’s part.

@edolstra

This comment has been minimized.

Copy link
Member

@edolstra edolstra commented Jun 6, 2019

Yes, you can jump through hoops and get a writable system volume, but only until the next reboot, so this is not a useful option.

Maybe you can use this to create a /nix symlink to another volume, which would hopefully persist across reboots?

@mroi

This comment has been minimized.

Copy link
Author

@mroi mroi commented Jun 6, 2019

That was my first try to get it temporarily working again. But nix-env complains then:

error: cannot open connection to remote store 'daemon': the path '/nix' is a symlink; this is not allowed for the Nix store and its parent directories

If that error can be safely ignored, I think this would be a workaround worth exploring. The way I understand things, anything you place in the root folder would persist until the next system update.

@LnL7

This comment has been minimized.

Copy link
Member

@LnL7 LnL7 commented Jun 6, 2019

There's no real reason for the nix store to be on the system volume, with apfs it's easy to create a separate volume and mount it to /nix. But perhaps there's something I'm missing because I don't understand the need for these firmlinks, the equivalent is possible with mounts similar to how the store is mounted readonly on NixOS.

@edolstra

This comment has been minimized.

Copy link
Member

@edolstra edolstra commented Jun 6, 2019

@mroi You can set NIX_IGNORE_SYMLINK_STORE=1 to disable that check. (This should probably be turned into a nix.conf option.)

@jwiegley

This comment has been minimized.

Copy link

@jwiegley jwiegley commented Jun 6, 2019

@edolstra Great idea, added #2926.

@mroi

This comment has been minimized.

Copy link
Author

@mroi mroi commented Jun 6, 2019

@edolstra Thanks for the tip, this is a useful workaround.

But future Nix-users on macOS still need to find a reliable way to inject this symlink into the read-only root directory. This probably involves some advanced steps like rebooting into recovery mode after every system upgrade.

@LnL7 The same would be true for /nix as a mount point, because we would need to create this directory on the read-only volume.

@LnL7

This comment has been minimized.

Copy link
Member

@LnL7 LnL7 commented Jun 6, 2019

Sure but if Apple doesn't want to provide some sort of solution for one of these cases there will be no alternative to changing the default prefix. Which is arguably is not worth it for a secondary platform that's continuously moving further away from UNIX.

Is there an overview of the full volume layout somewhere? /usr/local and /Users are used as examples but I can't imagine that's everything.

@kevingriffin

This comment has been minimized.

Copy link

@kevingriffin kevingriffin commented Jun 7, 2019

@mroi How have you been testing this? In my testing, I was able to install nix, but I assumed it was due to the comment in the presentation that "system volume is writeable in the developer preview". I'd also like to help test, but was able to get nix installed:

kevingriffin@KevinnoiMac /nix % uname -a
Darwin KevinnoiMac.local 19.0.0 Darwin Kernel Version 19.0.0: Fri May 24 17:36:10 PDT 2019; root:xnu-6041.0.0.111.5~1/RELEASE_X86_64 x86_64
kevingriffin@KevinnoiMac /nix % ls /nix
store	var
kevingriffin@KevinnoiMac /nix % nix-env -iA nixpkgs.hello
installing 'hello-2.10'
these paths will be fetched (0.02 MiB download, 0.07 MiB unpacked):
  /nix/store/c4w9z1kzzkdsgvgr6cy9ggl39s2yzn70-hello-2.10
copying path '/nix/store/c4w9z1kzzkdsgvgr6cy9ggl39s2yzn70-hello-2.10' from 'https://cache.nixos.org'...
building '/nix/store/iwcsfsh5hxyzpymnkp1r54nb3zzpd7mg-user-environment.drv'...
created 2 symlinks in user environment```
@matthewbauer

This comment has been minimized.

Copy link
Member

@matthewbauer matthewbauer commented Jun 7, 2019

It looks like you can still access the /nix directory after the update, it's just in /System/Volumes/Data/nix. Too bad there's no chroot equivalent on macOS.

@matthewbauer

This comment has been minimized.

Copy link
Member

@matthewbauer matthewbauer commented Jun 7, 2019

sudo mount -uw / && sudo ln -s /System/Volumes/Data/nix /nix works for me! This is on SIP even! Having the installer handle this might work.

@kevingriffin

This comment has been minimized.

Copy link

@kevingriffin kevingriffin commented Jun 7, 2019

I wonder if that will continue to work in the later builds, when they plan to enforce the read-only aspect.

@mroi

This comment has been minimized.

Copy link
Author

@mroi mroi commented Jun 7, 2019

Apple says in the WWDC talk, that the first beta seeds mount the system volume read-write, but this is only temporary. A later seed will turn it read-only. You can create a file /.rootro to opt-in to the read-only mount now.

@mroi

This comment has been minimized.

Copy link
Author

@mroi mroi commented Jun 7, 2019

But there is good news. I talked to an Apple engineer on Twitter and they are working on a solution to create symlinks in the root directory even when the system volume is mounted read-only. I guess we’ll just wait and see how this turns out.

For now, symlinking /nix to /System/Volumes/Data/nix in combination with NIX_IGNORE_SYMLINK_STORE=1 is a viable workaround for everyone on the macOS Catalina testing train.

@angerman

This comment has been minimized.

Copy link

@angerman angerman commented Jul 6, 2019

@grahamc remember our discussion about symlink /nix and the installer during ZuriHac? Turns out, if I had read the issue tracker more closely I had known about NIX_IGNORE_SYMLINK_STORE=1, and wouldn't have been as annoyed.

@matthewbauer

This comment has been minimized.

Copy link
Member

@matthewbauer matthewbauer commented Jul 10, 2019

This appears to be fixed in 10.15 Beta 3! I can create a /nix directory and use it as a store, while /System, /bin, /sbin, etc. are still read only. Users upgrading from previous beta versions can do:

sudo rm -f /nix
sudo mv /System/Volumes/Data/nix /nix

then start the daemon if you are in multi user:

sudo launchctl load /Library/LaunchDaemons/org.nixos.nix-daemon.plist
sudo launchctl start org.nixos.nix-daemon

New installs should work out of the box.

@EHYPERCHICKEN

This comment has been minimized.

Copy link

@EHYPERCHICKEN EHYPERCHICKEN commented Jul 10, 2019

/ will ship as non-writeable -- what you're seeing is a known issue where it can become writeable due to another system component. If you reboot, you will not see this work.

@matthewbauer

This comment has been minimized.

Copy link
Member

@matthewbauer matthewbauer commented Jul 12, 2019

/ will ship as non-writeable -- what you're seeing is a known issue where it can become writeable due to another system component. If you reboot, you will not see this work.

Will sudo mount -uw / continue to work?

@EHYPERCHICKEN

This comment has been minimized.

Copy link

@EHYPERCHICKEN EHYPERCHICKEN commented Jul 12, 2019

Will sudo mount -uw / continue to work?

If you disable SIP, yes.

@matthewbauer

This comment has been minimized.

Copy link
Member

@matthewbauer matthewbauer commented Jul 12, 2019

Will sudo mount -uw / continue to work?

If you disable SIP, yes.

Ugh, I don't think we want to recommend that. Are any exceptions for Nix possible? I saw that Fink's /sw is included in root now, if we could get something like this for Nix it would be extremely helpful.

To be clear, changing the root directory of Nix from /nix to something else is possible, but also very painful. We would lose some resource sharing abilities between macOS & Linux, as well as require setting up a special binary cache. The only alternative to this is recommending to our users insecure practices like disabling SIP.

@EHYPERCHICKEN

This comment has been minimized.

Copy link

@EHYPERCHICKEN EHYPERCHICKEN commented Jul 12, 2019

We're working on something that will allow you to create either a symlink or empty directory at / even though it's read-only. So you could go one of two ways:

  1. Create /nix symlink to the data volume
  2. Put the repo on a separate apfs volume that space shares within the container, create a /nix empty directory, and then mount that volume on that directory
@matthewbauer

This comment has been minimized.

Copy link
Member

@matthewbauer matthewbauer commented Jul 12, 2019

We're working on something that will allow you to create either a symlink or empty directory at / even though it's read-only. So you could go one of two ways:

  1. Create /nix symlink to the data volume
  2. Put the repo on a separate apfs volume that space shares within the container, create a /nix empty directory, and then mount that volume on that directory

Sounds reasonable! The main concern with (1) is that realpath(3) will still report /System/Volumes/Data/nix when pwd=/nix (#2926). That can get /System/Volumes/Data/nix hardcoded for some build systems. This could be a problem when that directory exists on Catalina but not on other machines. We need something a little bit stronger than a symlink ;)

@mroi

This comment has been minimized.

Copy link
Author

@mroi mroi commented Jul 15, 2019

If we were to mount Nix from a separate volume, we could use the opportunity to use the case-sensitive version of APFS. This would remove the need to fix weirdo packages relying on files that just differ in case.

@EHYPERCHICKEN

This comment has been minimized.

Copy link

@EHYPERCHICKEN EHYPERCHICKEN commented Jul 15, 2019

I never object to a change which spreads the case-sensitivity gospel.

@angerman

This comment has been minimized.

Copy link

@angerman angerman commented Jul 15, 2019

If we were to mount Nix from a separate volume, we could use the opportunity to use the case-sensitive version of APFS. This would remove the need to fix weirdo packages relying on files that just differ in case.

I have been doing this with great success (although with hfs+ case sensitive) with an external ssd for a while now. The primary motivation being, that I can take my nix store from my desktop with me when I use my laptop while walking about.

@chepner

This comment has been minimized.

Copy link

@chepner chepner commented Jan 9, 2020

Just wanted to stay I wish I could use NIX without all these hoops. It seems like a promising package manager.

It's a one-time hurdle, at least. Using nix works as well as it did before; it's just the initial installation that people are working to adapt to Catalina.

@mroi

This comment has been minimized.

Copy link
Author

@mroi mroi commented Jan 9, 2020

Still trying to figure out the auto-mount on startup.

Maybe this disk image autmount project of mine can help a little bit along the way.

@nycnewman

This comment has been minimized.

Copy link

@nycnewman nycnewman commented Jan 13, 2020

Minor update with plist file to go into /Library/LaunchDaemons/ to remount on startup

nix-volume.txt -> nix-volume.sh
com.acme.mount.nix.txt -> com..mount-nix.plist

nix-volume.txt
com.acme.mount-nix.txt

Seems to work. Still investigating possible performance implications.

@lilyball

This comment has been minimized.

Copy link
Member

@lilyball lilyball commented Jan 14, 2020

@nycnewman This approach means the disk isn't mounted prior to user login session setup. We already have that problem with an encrypted Nix partition, but an unencrypted partition is mounted prior to GUI app restoration, and there's a workaround using a login hook for forcing the encrypted partition to mount first. The DMG approach doesn't seem to support this at all, which is rather unfortunate as it causes problems with GUI app restoration depends on stuff installed from Nix (such as restoring Terminal.app when your login shell is from Nix).

@nycnewman

This comment has been minimized.

Copy link

@nycnewman nycnewman commented Jan 14, 2020

@lilyball Finding mixed messages regarding Login Hooks. Many references suggest this is deprecated and to use LaunchDaemons. If you have examples elsewhere, happy to try.

@lilyball

This comment has been minimized.

Copy link
Member

@lilyball lilyball commented Jan 14, 2020

The login hook appears to execute before the GUI session is initialized (or rather, before GUI apps are launched). I haven't tried it myself but that's what I infer based on people using it to force-mount the encrypted partition.

AFAIK LaunchDaemons do not block GUI app initialization.

@mroi

This comment has been minimized.

Copy link
Author

@mroi mroi commented Jan 14, 2020

Finding mixed messages regarding Login Hooks. Many references suggest this is deprecated and to use LaunchDaemons.

Apple states login hook deprecation officially. However, adding a login script using configuration profiles is not deprecated.

@Feuermurmel

This comment has been minimized.

Copy link

@Feuermurmel Feuermurmel commented Jan 21, 2020

I think this issue has already been beaten to death and as a disclaimer, I haven't read all prior comments. I was planning on evaluating nix to use as the base to solve cross-platform packaging issues. But this issue seems like a real blocker as it would make the initial setup process harder not easier, when building on top of nix. :(

I can't understand why the nix store isn't just moved to /usr/local on all platforms. AFAIK that is the intended location for non-distribution-managed software on most Unix-like platforms.

@iain-henderson

This comment has been minimized.

Copy link

@iain-henderson iain-henderson commented Jan 21, 2020

I think this issue has already been beaten to death and as a disclaimer, I haven't read all prior comments. I was planning on evaluating nix to use as the base to solve cross-platform packaging issues. But this issue seems like a real blocker as it would make the initial setup process harder not easier, when building on top of nix. :(

I can't understand why the nix store isn't just moved to /usr/local on all platforms. AFAIK that is the intended location for non-distribution-managed software on most Unix-like platforms.

That would conform to the Filesystem Hierarchy Standard:
The /usr/local hierarchy is for use by the system administrator when installing software locally. It needs to be safe from being overwritten when the system software is updated. It may be used for programs and data that are shareable amongst a group of hosts, but not found in /usr.

@amrox

This comment has been minimized.

Copy link

@amrox amrox commented Jan 21, 2020

IMHO /opt/nix is a little cleaner that /usr/local/nix. Many pieces of software install to /usr/local already, and homebrew "owns" /usr/local on macOS.

By my reading, it seems like nix would qualify has "Add-on application software packages" in the Filesystem Hierarchy Standard . It doesn't seem like nix is registered as a LANANA provider, but it could be?

@chepner

This comment has been minimized.

Copy link

@chepner chepner commented Jan 21, 2020

Changing the store path is certainly possible, but at a cost: build products have the store path embedded in them, so artifacts that use something other than /nix aren't compatible with existing stores that do use /nix (including the upstream binary caches). Migrating an existing store is not as simple as mv /nix /usr/local/.

@emptyflask

This comment has been minimized.

Copy link

@emptyflask emptyflask commented Jan 21, 2020

I think this issue has already been beaten to death and as a disclaimer, I haven't read all prior comments. I was planning on evaluating nix to use as the base to solve cross-platform packaging issues. But this issue seems like a real blocker as it would make the initial setup process harder not easier, when building on top of nix. :(

I can't understand why the nix store isn't just moved to /usr/local on all platforms. AFAIK that is the intended location for non-distribution-managed software on most Unix-like platforms.

Except for NixOS...

@alexvorobiev

This comment has been minimized.

Copy link

@alexvorobiev alexvorobiev commented Jan 21, 2020

IMHO /opt/nix is a little cleaner that /usr/local/nix

I agree. /opt/nix seems a natural location for all platforms. In an enterprise environment where there are tons of non nix-aware automation/inventory/security tools running all the time installing something under /opt is much much easier than /.

@clhodapp

This comment has been minimized.

Copy link

@clhodapp clhodapp commented Jan 22, 2020

I can't understand why the nix store isn't just moved to /usr/local on all platforms

The problem with moving the nix store at all is that it breaks all the running systems out in the world. Moving the store on a system effectively amounts to installing a new OS. This is because absolute store paths are baked into the content of most packages. So... if you mv the store, you break all your existing packages. If you install a package that expects the store to be in a different place, it won't work on your system.

That said, it's definitely a shame that /nix was put at the root of the filesystem in the first place.

@7c6f434c

This comment has been minimized.

Copy link
Member

@7c6f434c 7c6f434c commented Jan 22, 2020

@emptyflask

This comment has been minimized.

Copy link

@emptyflask emptyflask commented Jan 22, 2020

I use Nix on MacOS on my work computer (and NixOS at home), but I think changing /nix to /opt/nix or some other location is a mistake. It looks as though there's already a workaround for this new arbitrary restriction that Apple created, so let's just test that thoroughly and document it. If installing Nix needs to be as simple as something like Homebrew, it's possible to include this extra setup in an install script.

@lilyball

This comment has been minimized.

Copy link
Member

@lilyball lilyball commented Jan 22, 2020

@emptyflask The particular workaround you linked to is the symlink approach, which appears to break a lot of packages. I strongly recommend using the separate volume approach instead (documented many times in this thread).

@Renha

This comment has been minimized.

Copy link

@Renha Renha commented Jan 22, 2020

I’ll wait until one of approaches would be implemented in the installer

@Feuermurmel

This comment has been minimized.

Copy link

@Feuermurmel Feuermurmel commented Jan 22, 2020

I’ll wait until one of approaches would be implemented in the installer

Yeah, that's my take on this too. I better wait this out until a solution has been implemented that I can evaluate.

@charles-dyfis-net

This comment has been minimized.

Copy link

@charles-dyfis-net charles-dyfis-net commented Jan 22, 2020

I've deployed the solution in my gist (configuring a volume) in a corporate environment. Personally, I'd call it something already implemented, and available for evaluation.

@lilyball

This comment has been minimized.

Copy link
Member

@lilyball lilyball commented Jan 22, 2020

I've been running with the separate volume approach for a while now with zero problems (though I keep the volume unencrypted so it mounts prior to GUI initialization; I don't use it to build work source, just for tooling support). As I understand it the only real holdup with getting this into the installer is testing it on a wide variety of preconditions.

@truh

This comment has been minimized.

Copy link

@truh truh commented Jan 23, 2020

@emptyflask The particular workaround you linked to is the symlink approach, which appears to break a lot of packages. I strongly recommend using the separate volume approach instead (documented many times in this thread).

I have a separate volume mounted with an entry in synthetic.conf to /nix, /nix is still a link to Volumes/NixStore (NixStore is the name of the volume). Is there a way to mount the volume directly to /nix?

@lilyball

This comment has been minimized.

Copy link
Member

@lilyball lilyball commented Jan 23, 2020

@truh The instructions are higher up in this thread, but basically, your /etc/synthetic.conf should have a line just consisting of /nix (nothing else), your volume should have the label "Nix", and your /etc/fstab should have a line instructing the system to mount that volume at /nix. I don't remember the precise formulation of the line, but it's in this thread somewhere.

With these instructions, if your volume is encrypted then GUI apps will start to restore before it finishes mounting (this is a problem if Terminal.app is one of these apps and you use a Nix-installed shell as your login shell). If your volume is unencrypted then it will mount prior to the GUI session logging in. That said, I assume this caveat applies to your current setup as well (where you're symlinking to /Volumes/NixStore).

@lionello

This comment has been minimized.

Copy link

@lionello lionello commented Jan 28, 2020

This worked for me: #2925 (comment)

@keithy

This comment has been minimized.

Copy link

@keithy keithy commented Feb 1, 2020

Another data point: Fedora Core OS also locks down / and doesn't provide a work around, thus forcing nix users to work via a chroot.

@matthewbauer

This comment has been minimized.

Copy link
Member

@matthewbauer matthewbauer commented Feb 3, 2020

Another data point: Fedora Core OS also locks down / and doesn't provide a work around, thus forcing nix users to work via a chroot.

Maybe we should make a separate ticket for this. But it looks like you can make it work with some overlays:

https://www.mikenowak.org/working-around-read-file-systems-coreos-overlay/

(replacing /opt with /nix)

@keithy

This comment has been minimized.

Copy link

@keithy keithy commented Feb 3, 2020

My understanding is that at the very least you would need a mountpoint. My current plan is to provide a login shell that has a chroot with everything in place.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.