Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

/nix will not be writable on macOS Catalina #2925

Closed
mroi opened this issue Jun 6, 2019 · 339 comments
Closed

/nix will not be writable on macOS Catalina #2925

mroi opened this issue Jun 6, 2019 · 339 comments

Comments

@mroi
Copy link

@mroi mroi commented Jun 6, 2019

This is not a short term bug, but it will become an issue when macOS Catalina is released this fall. macOS is now split across two volumes (system and data) with a read-only system volume. This means that /nix will no longer be writable.

Some more information can be found in the related WWDC talk and some session notes people took from a Q&A.

Summary: the system volume, which is mounted at / will become non-writable. Some directories that need to be writable are connected via firmlinks (an Apple invention) to the data volume. /nix is not among these locations, so with the release of macOS Catalina, this location is no longer an option.

I see two possible solutions:

  1. I could try and file a bug to convince Apple to pre-install /nix as a firmlink to a writable location. I think this has limited success and Nix would then depend on Apple to not drop this link in a future release.
  2. We could move Nix on macOS to a different default location. Possible locations are those that Apple chooses to pre-install as links to writable locations. Two examples are /usr/local and /opt, so we could move to /usr/local/nix or /opt/nix. I would hope that these locations are common enough so that Apple would not drop them in the future.

I wanted to raise this issue early, before it becomes a problem for users. If this issue tracker is not the right place, please feel free to move this discussion elsewhere. I would also be available for testing any potential solution, since I have access to a macOS Catalina beta.

@edolstra
Copy link
Member

@edolstra edolstra commented Jun 6, 2019

Realistically, this may force us to drop support for macOS. Using a different store location for macOS would require a separate binary cache and a separate Hydra instance, and there would be no guarantee that the new location wouldn't break in the future.

I did notice in the talk that it's still possible to make the system volume writable, though not in a persistent way.

Loading

@mroi
Copy link
Author

@mroi mroi commented Jun 6, 2019

Speaking for myself: I would be quite sad if macOS support would go away.
I will file a bug with Apple and see what happens. Cannot hurt.

Yes, you can jump through hoops and get a writable system volume, but only until the next reboot, so this is not a useful option.

Loading

@jwiegley
Copy link

@jwiegley jwiegley commented Jun 6, 2019

@edolstra Before canceling support, we can ask the community if it's willing to host its own binary cache and Hydra instance, just as we've enlisted a host of volunteers to keep nixpkgs at near parity on Darwin. Having the answer be, "This doesn't easily fit our model so we won't support Mac" would be an unfortunate response.

Loading

@mroi
Copy link
Author

@mroi mroi commented Jun 6, 2019

I think no such decision should be made lightly, but it is definitely a drastic change on Apple’s part.

Loading

@edolstra
Copy link
Member

@edolstra edolstra commented Jun 6, 2019

Yes, you can jump through hoops and get a writable system volume, but only until the next reboot, so this is not a useful option.

Maybe you can use this to create a /nix symlink to another volume, which would hopefully persist across reboots?

Loading

@mroi
Copy link
Author

@mroi mroi commented Jun 6, 2019

That was my first try to get it temporarily working again. But nix-env complains then:

error: cannot open connection to remote store 'daemon': the path '/nix' is a symlink; this is not allowed for the Nix store and its parent directories

If that error can be safely ignored, I think this would be a workaround worth exploring. The way I understand things, anything you place in the root folder would persist until the next system update.

Loading

@LnL7
Copy link
Member

@LnL7 LnL7 commented Jun 6, 2019

There's no real reason for the nix store to be on the system volume, with apfs it's easy to create a separate volume and mount it to /nix. But perhaps there's something I'm missing because I don't understand the need for these firmlinks, the equivalent is possible with mounts similar to how the store is mounted readonly on NixOS.

Loading

@edolstra
Copy link
Member

@edolstra edolstra commented Jun 6, 2019

@mroi You can set NIX_IGNORE_SYMLINK_STORE=1 to disable that check. (This should probably be turned into a nix.conf option.)

Loading

@jwiegley
Copy link

@jwiegley jwiegley commented Jun 6, 2019

@edolstra Great idea, added #2926.

Loading

@mroi
Copy link
Author

@mroi mroi commented Jun 6, 2019

@edolstra Thanks for the tip, this is a useful workaround.

But future Nix-users on macOS still need to find a reliable way to inject this symlink into the read-only root directory. This probably involves some advanced steps like rebooting into recovery mode after every system upgrade.

@LnL7 The same would be true for /nix as a mount point, because we would need to create this directory on the read-only volume.

Loading

@LnL7
Copy link
Member

@LnL7 LnL7 commented Jun 6, 2019

Sure but if Apple doesn't want to provide some sort of solution for one of these cases there will be no alternative to changing the default prefix. Which is arguably is not worth it for a secondary platform that's continuously moving further away from UNIX.

Is there an overview of the full volume layout somewhere? /usr/local and /Users are used as examples but I can't imagine that's everything.

Loading

@kevingriffin
Copy link

@kevingriffin kevingriffin commented Jun 7, 2019

@mroi How have you been testing this? In my testing, I was able to install nix, but I assumed it was due to the comment in the presentation that "system volume is writeable in the developer preview". I'd also like to help test, but was able to get nix installed:

kevingriffin@KevinnoiMac /nix % uname -a
Darwin KevinnoiMac.local 19.0.0 Darwin Kernel Version 19.0.0: Fri May 24 17:36:10 PDT 2019; root:xnu-6041.0.0.111.5~1/RELEASE_X86_64 x86_64
kevingriffin@KevinnoiMac /nix % ls /nix
store	var
kevingriffin@KevinnoiMac /nix % nix-env -iA nixpkgs.hello
installing 'hello-2.10'
these paths will be fetched (0.02 MiB download, 0.07 MiB unpacked):
  /nix/store/c4w9z1kzzkdsgvgr6cy9ggl39s2yzn70-hello-2.10
copying path '/nix/store/c4w9z1kzzkdsgvgr6cy9ggl39s2yzn70-hello-2.10' from 'https://cache.nixos.org'...
building '/nix/store/iwcsfsh5hxyzpymnkp1r54nb3zzpd7mg-user-environment.drv'...
created 2 symlinks in user environment```

Loading

@matthewbauer
Copy link
Member

@matthewbauer matthewbauer commented Jun 7, 2019

It looks like you can still access the /nix directory after the update, it's just in /System/Volumes/Data/nix. Too bad there's no chroot equivalent on macOS.

Loading

@matthewbauer
Copy link
Member

@matthewbauer matthewbauer commented Jun 7, 2019

sudo mount -uw / && sudo ln -s /System/Volumes/Data/nix /nix works for me! This is on SIP even! Having the installer handle this might work.

Loading

@kevingriffin
Copy link

@kevingriffin kevingriffin commented Jun 7, 2019

I wonder if that will continue to work in the later builds, when they plan to enforce the read-only aspect.

Loading

@mroi
Copy link
Author

@mroi mroi commented Jun 7, 2019

Apple says in the WWDC talk, that the first beta seeds mount the system volume read-write, but this is only temporary. A later seed will turn it read-only. You can create a file /.rootro to opt-in to the read-only mount now.

Loading

@mroi
Copy link
Author

@mroi mroi commented Jun 7, 2019

But there is good news. I talked to an Apple engineer on Twitter and they are working on a solution to create symlinks in the root directory even when the system volume is mounted read-only. I guess we’ll just wait and see how this turns out.

For now, symlinking /nix to /System/Volumes/Data/nix in combination with NIX_IGNORE_SYMLINK_STORE=1 is a viable workaround for everyone on the macOS Catalina testing train.

Loading

@angerman
Copy link

@angerman angerman commented Jul 6, 2019

@grahamc remember our discussion about symlink /nix and the installer during ZuriHac? Turns out, if I had read the issue tracker more closely I had known about NIX_IGNORE_SYMLINK_STORE=1, and wouldn't have been as annoyed.

Loading

@matthewbauer
Copy link
Member

@matthewbauer matthewbauer commented Jul 10, 2019

This appears to be fixed in 10.15 Beta 3! I can create a /nix directory and use it as a store, while /System, /bin, /sbin, etc. are still read only. Users upgrading from previous beta versions can do:

sudo rm -f /nix
sudo mv /System/Volumes/Data/nix /nix

then start the daemon if you are in multi user:

sudo launchctl load /Library/LaunchDaemons/org.nixos.nix-daemon.plist
sudo launchctl start org.nixos.nix-daemon

New installs should work out of the box.

Loading

@EHYPERCHICKEN
Copy link

@EHYPERCHICKEN EHYPERCHICKEN commented Jul 10, 2019

/ will ship as non-writeable -- what you're seeing is a known issue where it can become writeable due to another system component. If you reboot, you will not see this work.

Loading

@matthewbauer
Copy link
Member

@matthewbauer matthewbauer commented Jul 12, 2019

/ will ship as non-writeable -- what you're seeing is a known issue where it can become writeable due to another system component. If you reboot, you will not see this work.

Will sudo mount -uw / continue to work?

Loading

@EHYPERCHICKEN
Copy link

@EHYPERCHICKEN EHYPERCHICKEN commented Jul 12, 2019

Will sudo mount -uw / continue to work?

If you disable SIP, yes.

Loading

@matthewbauer
Copy link
Member

@matthewbauer matthewbauer commented Jul 12, 2019

Will sudo mount -uw / continue to work?

If you disable SIP, yes.

Ugh, I don't think we want to recommend that. Are any exceptions for Nix possible? I saw that Fink's /sw is included in root now, if we could get something like this for Nix it would be extremely helpful.

To be clear, changing the root directory of Nix from /nix to something else is possible, but also very painful. We would lose some resource sharing abilities between macOS & Linux, as well as require setting up a special binary cache. The only alternative to this is recommending to our users insecure practices like disabling SIP.

Loading

@EHYPERCHICKEN
Copy link

@EHYPERCHICKEN EHYPERCHICKEN commented Jul 12, 2019

We're working on something that will allow you to create either a symlink or empty directory at / even though it's read-only. So you could go one of two ways:

  1. Create /nix symlink to the data volume
  2. Put the repo on a separate apfs volume that space shares within the container, create a /nix empty directory, and then mount that volume on that directory

Loading

@matthewbauer
Copy link
Member

@matthewbauer matthewbauer commented Jul 12, 2019

We're working on something that will allow you to create either a symlink or empty directory at / even though it's read-only. So you could go one of two ways:

  1. Create /nix symlink to the data volume
  2. Put the repo on a separate apfs volume that space shares within the container, create a /nix empty directory, and then mount that volume on that directory

Sounds reasonable! The main concern with (1) is that realpath(3) will still report /System/Volumes/Data/nix when pwd=/nix (#2926). That can get /System/Volumes/Data/nix hardcoded for some build systems. This could be a problem when that directory exists on Catalina but not on other machines. We need something a little bit stronger than a symlink ;)

Loading

@mroi
Copy link
Author

@mroi mroi commented Jul 15, 2019

If we were to mount Nix from a separate volume, we could use the opportunity to use the case-sensitive version of APFS. This would remove the need to fix weirdo packages relying on files that just differ in case.

Loading

@EHYPERCHICKEN
Copy link

@EHYPERCHICKEN EHYPERCHICKEN commented Jul 15, 2019

I never object to a change which spreads the case-sensitivity gospel.

Loading

@angerman
Copy link

@angerman angerman commented Jul 15, 2019

If we were to mount Nix from a separate volume, we could use the opportunity to use the case-sensitive version of APFS. This would remove the need to fix weirdo packages relying on files that just differ in case.

I have been doing this with great success (although with hfs+ case sensitive) with an external ssd for a while now. The primary motivation being, that I can take my nix store from my desktop with me when I use my laptop while walking about.

Loading

@lunaticare
Copy link

@lunaticare lunaticare commented May 6, 2020

I use this script to create volumes in semi-automated fashion with FileVault enabled:

export CRYPTO_USER=$(sudo dscl . read /Users/$(whoami) | grep 'GeneratedUID' | cut -d ' ' -f2)
export PASSPHRASE=$(pwgen -C 20)

/nix volume:

sudo diskutil apfs addVolume disk1 APFSX Nix -mountpoint /nix
sudo diskutil enableOwnership /nix
echo $PASSPHRASE | sudo diskutil apfs encrypt /nix -user disk -stdinpassphrase
echo $PASSPHRASE | sudo diskutil apfs encrypt /nix -user $CRYPTO_USER  -stdinpassphrase
sudo chflags hidden /nix
echo 'LABEL=Nix /nix apfs rw' tee -a /etc/fstab
UUID=$(diskutil info -plist /nix | plutil -extract VolumeUUID xml1 - -o - | plutil -p - | sed -e 's/"//g')
security add-generic-password -l Nix -a "$UUID" -s "$UUID" -D "Encrypted Volume Password" -w "$PASSPHRASE" \
 -T "/System/Library/CoreServices/APFSUserAgent" -T "/System/Library/CoreServices/CSUserAgent"

Same for /run:

sudo diskutil apfs addVolume disk1 APFSX Run -mountpoint /run
sudo diskutil enableOwnership /run
echo $PASSPHRASE | sudo diskutil apfs encrypt /run -user disk -stdinpassphrase
echo $PASSPHRASE | sudo diskutil apfs encrypt /run -user $CRYPTO_USER -stdinpassphrase
sudo chflags hidden /run
# sudo chown -R <username> /run
echo 'LABEL=Run /run apfs rw' | sudo tee -a /etc/fstab
UUID=$(diskutil info -plist /run | plutil -extract VolumeUUID xml1 - -o - | plutil -p - | sed -e 's/"//g')
security add-generic-password -l Run -a "$UUID" -s "$UUID" -D "Encrypted Volume Password" -w "$PASSPHRASE" \
 -T "/System/Library/CoreServices/APFSUserAgent" -T "/System/Library/CoreServices/CSUserAgent"

Loading

@johanatan
Copy link

@johanatan johanatan commented May 6, 2020

This is pretty complicated just for trying to install a package manager. Any ETA on when the installer instructions themselves will be "fixed"? I don't necessarily want to dive into all of these details and would prefer something that "just works".

Loading

@druimalban
Copy link

@druimalban druimalban commented May 6, 2020

It may be a bit complicated but it is not exactly helped by people continuing to post the same things over and over again. Some comment (now swallowed up, never to be seen again) foreshadowed exactly this and warned against it

Loading

@johanatan
Copy link

@johanatan johanatan commented May 6, 2020

Yes, someone predicted this was going to be a problem with the OS update that was released months ago. They even did so during the "beta" phase which offered plenty of time to get ahead of the problem and fix the installation at the main landing page where people from all over the world are continually landing to this day. :)

Loading

@druimalban
Copy link

@druimalban druimalban commented May 6, 2020

No, it was this comment: #2925 (comment)

and it was right

Loading

@johanatan
Copy link

@johanatan johanatan commented May 6, 2020

I was thinking of a different (and earlier one) obviously. Perhaps on a completely different issue that I went through before landing here.

Loading

@johanatan
Copy link

@johanatan johanatan commented May 6, 2020

And that's still ridiculously complicated. I would say that your software project is failing if it requires reading a wall of text and issuing lots of volume-based commands just to get a basic installation complete. Major fail in fact.

Loading

@lunaticare
Copy link

@lunaticare lunaticare commented May 6, 2020

@druimalban @johanatan will it be sane to offer interactive script to setup raw/encrypted volumes for a single user installation?
What a good solution might look like?

Loading

@alper
Copy link

@alper alper commented May 6, 2020

Unsubscribing here for the time being. Somebody let me know when this is usable for a casual macOS user because this does not seem like it.

Loading

@abathur
Copy link
Member

@abathur abathur commented May 6, 2020

This limbo is frustrating from every angle, but we won't get anywhere good by venting at people (even, I suppose, at Apple?)

Having enjoyed using Nix on macOS over the past 2+ years, it's frustrating to see this keep away prospective users. That said, it's also abnormal for a "basic installation" to have to go create new volumes (let alone encrypted ones) and I think we'd be in an even worse place if the installer was out in the wild doing it in a half-baked way.

Loading

@johanatan
Copy link

@johanatan johanatan commented May 6, 2020

This installer is out in the wild doing things in a half-baked way. I have no idea the state of my system in fact nor do I have any way to completely erase whatever Nix did to it. /nix remains undeletable and God knows what else Nix did before it crashed.

Loading

@abathur
Copy link
Member

@abathur abathur commented May 6, 2020

Did you follow the removal steps you quoted in #2925 (comment)?

Loading

@johanatan
Copy link

@johanatan johanatan commented May 6, 2020

Not yet. That's the first I've heard of any "removal steps". I didn't read that comment in detail yet--just saw a wall of text and said "that'll have to wait until later".

Loading

@samuela
Copy link
Member

@samuela samuela commented May 7, 2020

What's the status on this issue? The website and installer are quite misleading in the sense that they still claim complete compatibility with macOS. Is there a path forward to using Nix on macOS without any hacks?

Loading

@wmertens
Copy link
Contributor

@wmertens wmertens commented May 8, 2020

@samuela macOS does not allow simply creating /nix. So that needs to be configured in some way, not really a "hack".

The easiest way to have Nix on macOS would be if Nix used /opt/nix or /usr/local/nix instead of /nix.

Loading

@samuela
Copy link
Member

@samuela samuela commented May 8, 2020

So what is the problem with /opt/nix or /usr/local/nix/ instead of /nix? This seems like a real showstopper bug...

Loading

@wmertens
Copy link
Contributor

@wmertens wmertens commented May 8, 2020

@samuela it means either doubling the storage for Hydra, or changing ALL the Nix installs everywhere.

One hacky option is to rewrite store paths as they get downloaded. So any mention of /nix/store/...name would be rewritten in the file to /opt/nix/...(name-4chars). This would work for most of the files, but still needs support in nix-store and all over nixpkgs where /nix is hardcoded

Loading

@alper
Copy link

@alper alper commented May 8, 2020

@wmertens How much storage would get doubled?

Loading

@wmertens
Copy link
Contributor

@wmertens wmertens commented May 8, 2020

@alper any package that Hydra produces would have to be produced twice, once for /nix and once for /opt/nix.

It's not quite a doubling, because not all packages are built for Darwin and files that don't reference store paths would likely be the same and can be hardlinked.

I don't know how big the Hydra storage is.

Loading

@samuela
Copy link
Member

@samuela samuela commented May 8, 2020

Seeing as this issue has been open since 2019, it seems as though macOS is no longer a supported platform. Is that true? Or is there a roadmap to fixing this?

Loading

@alper
Copy link

@alper alper commented May 8, 2020

Because shouldn’t storage be a non-issue at this point in history?

Loading

@grahamc
Copy link
Member

@grahamc grahamc commented May 8, 2020

I'm going to lock this until we're able to get a fix out. Thanks for everybody's contributions and help in figuring out this problem, hopefully it is soon.

Loading

@NixOS NixOS locked as too heated and limited conversation to collaborators May 8, 2020
@edolstra
Copy link
Member

@edolstra edolstra commented May 8, 2020

@wmertens We wouldn't need to produce packages twice, since we'd have one Hydra instance (and binary cache) for macOS (using the /opt prefix) and another instance for all other platforms.

The main consequence of using a separate prefix for macOS is that you can't have Hydra jobsets anymore containing jobs for macOS and Linux. It would also make it harder to deploy from macOS to Linux.

Loading

@domenkozar domenkozar pinned this issue May 19, 2020
@domenkozar
Copy link
Member

@domenkozar domenkozar commented May 26, 2020

Documentation for installation of Nix on Catalina is available.

Note that until Nix 2.3.5 release is out, the instructions won't work. PR

Loading

@domenkozar
Copy link
Member

@domenkozar domenkozar commented May 27, 2020

Nix 2.3.5 has been released with Catalina fix!

Loading

@domenkozar domenkozar unpinned this issue May 28, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.