Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rootless Nix: Path Rewriting (would like to contribute) #1971

Open
timjrd opened this issue Mar 12, 2018 · 29 comments
Open

Rootless Nix: Path Rewriting (would like to contribute) #1971

timjrd opened this issue Mar 12, 2018 · 29 comments
Assignees

Comments

@timjrd
Copy link

@timjrd timjrd commented Mar 12, 2018

Hi :) .

I have been using and playing with Nix for quite some time now, and as I find it a very clever piece of software, I am looking for a way to contribute.

Recently I had to install Nix on a university workstation where I couldn't create /nix. As I still wanted to use the official binary cache, I tried the PRoot workaround - even though it degrades performances - but it didn't work well (obscure bugs that don't happen on my normal installation like failing mallocs).

Anyway, it seems that a lot of users would be very interested in a Nix feature allowing a rootless installation compatible with the official binary cache.

@edolstra proposed the solution to rewrite store prefixes by patching pre-built binaries when exchanging paths between stores with different prefixes.

To be compatible with the existing pre-built packages, the canonical store prefix (as stated by Eelco in the above comment) could be /nix/store/.

If the user's store prefix is shorter - say /a/b/ - we can rewrite /nix/store/ to /a/b///////. Both strings must have the exact same length so we use multiple successive slashes as padding as allowed by POSIX.

If the user's store prefix is longer - say /home/user/nix/store/ - we can rewrite /nix/store/ to say /tmp/x0z3i/ and then symlink /tmp/x0z3i/ to /home/user/nix/store/. As stated in the Nix manual, having the Nix store directory being a symlink is a bad idea because a builder could dereference that symlink, potentially resulting in /home/user/nix/store/ being present in the result of local builds. The solution would be to rewrite prefixes after each build, from /home/user/nix/store/ to /tmp/x0z3i/////////// in our example.

When exporting or sending paths/closures, we can rewrite /tmp/x0z3i/ to /nix/store/, with as many slashes after the prefix as needed.

If I have the consent of the core Nix developers, I would be very happy to start working on this feature during my summer break (I'm a computer science student). I'm writing this issue because I would like to discuss the relevance and the details of this possible feature with the developers ahead of time.

So, what do you think of this ?

@dtzWill
Copy link
Member

@dtzWill dtzWill commented Mar 13, 2018

If your university machines have new enough kernels (mine don't :P) you can use --store=/path/to/store' and Nix will use some bind-mount and mount namespace goodness to act as if that path was /nix/store from the perspective of builders and such. Particularly neat is that you can use nix runto enter a shell where/nix/store` is that path... which has worked pretty well for me on my own machines (as compared to the similar edge-case problems you describe with using proot).

@timjrd
Copy link
Author

@timjrd timjrd commented Mar 13, 2018

I was not aware of the new mount namespace feature, I just tested it on my machine and it works fine, but I see two drawbacks here: the installer still needs root access so you have to compile from source which is not very convenient, and more importantly some distributions including Debian disables user namespaces by default.

@edolstra
Copy link
Member

@edolstra edolstra commented Mar 13, 2018

The first issue (the installer needing to be installed in /nix) could be solved by creating a statically linked Nix.

Other than that, using chroot stores via --store ... seems a lot cleaner than doing path rewriting, which as you point out only works for paths shorter than /nix/store. Using indirections like /tmp/x0z3i opens a can of security worms since such symlinks could be deleted (e.g. on reboot) and recreated by another user.

@veprbl
Copy link
Member

@veprbl veprbl commented Mar 13, 2018

Both proot and --store require that you need to always be in chroot (run proot or nix-shell in all interactive shells, in crontab, in batch job system, all other users on a multiuser system would not be able to use anything without it too). proot is a bottleneck for most parallel tasks. I also had some stack allocation problems with it too, but they were solved by NixOS/nixpkgs@10cc95e . The main problem with --store is that it requires CLONE_NEWUSER and CLONE_NEWNS. They are disabled and/or limited to root in many distributions (e.g. RHEL7, Debian). There is a nice utility unshare(1) that allows to poke these.

I still think that it would be nice to have this feature. One could set up a private hydra cache to build in /xxxxxxxxxxxxxxxxxxxxxxxxxx/store/ (as many "x" as they need) and then it would be possible to rewrite resulting NAR's to many other different locations. Perhaps the first step would be to just check how well rewriting works using some external tool without trying to modify nix itself.

@7c6f434c
Copy link
Member

@7c6f434c 7c6f434c commented Mar 13, 2018

A limitation of path rewriting regardless of initial path length (and correspondingly a recommendation for choice of packages to test): if the path gets embedded into a JAR, or another ZIP-with-semantics, it can be more complicated to locate and rewrite it. I am not sure if this actually happens.

@cstrahan
Copy link
Contributor

@cstrahan cstrahan commented Mar 16, 2018

if the path gets embedded into a JAR, or another ZIP-with-semantics, it can be more complicated to locate and rewrite it

I do know that this happened with Bazel: NixOS/nixpkgs@c69d90b

@timjrd
Copy link
Author

@timjrd timjrd commented Mar 17, 2018

Following the suggestion of @veprbl I wrote a small tool to rewrite store prefixes in a binary stream (and in a NAR dump of a closure). I tested it on firefox and it seems to work well (firefox starts and seems to work normally). You can grab it here and test it that way :

make
./pack $(nix-build '<nixpkgs>' -A firefox) firefox.nar firefox.dep
./unpack /tmp/store firefox.nar firefox.dep 

@veprbl
Copy link
Member

@veprbl veprbl commented Mar 22, 2018

https://github.com/spack/spack supports binary package relocation:

They are even more optimistic and just use patchelf and install_name_tool on object files and plain text replace on text files.

@timjrd
Copy link
Author

@timjrd timjrd commented Mar 22, 2018

@veprbl Does this work for embedded constant strings like resources paths ? It seems to me that raw binary patching would be more robust and generic while being easier to implement (with the limitation of the path length of course).

@7c6f434c @cstrahan We could dive recursively into compressed archives and reconstruct them. Using something like libarchive it shouldn't be that hard. This could even be reused to improve Nix's automatic runtime dependencies detection ;-) .

@edolstra
Copy link
Member

@edolstra edolstra commented Mar 22, 2018

Compressed archives are not supported by Nix. They hide dependencies and don't work with hash rewriting. (I've been thinking about adding an option to enable hash rewriting for all builds, to catch builds that do hide dependencies.)

@copumpkin
Copy link
Member

@copumpkin copumpkin commented Mar 22, 2018

We've noodled in the past about pluggable rewriting though, to allow rewriting through more opaque formats. Might be fun to start exploring that, although I don't really see too much use for it outside of e.g., java JAR files (which will rarely contain paths anyway)

@shlevy
Copy link
Member

@shlevy shlevy commented Mar 22, 2018

💯 for pluggable rewriting (and pluggable reference registration...)

@edolstra
Copy link
Member

@edolstra edolstra commented Mar 22, 2018

Hell no. That means derivations will work on some Nix installations and fail in subtle ways on others, depending on their plugin configuration.

@shlevy
Copy link
Member

@shlevy shlevy commented Mar 22, 2018

I meant pluggable by the derivation. So a derivation can say "hey, if you need to rewrite paths, this is how you do it for me"

@copumpkin
Copy link
Member

@copumpkin copumpkin commented Mar 22, 2018

What @shlevy said 😄

@timjrd
Copy link
Author

@timjrd timjrd commented Mar 28, 2018

Anyone interested in testing this ?

Maybe I could properly rewrite this and add support for nested compressed archives with libarchive in a reasonably modular way to allow future extensions for other opaque formats. This could later be used in Nix to rewrite hashes in order to catch potentially hidden dependencies, or to rewrite store prefixes, or to improve automatic runtime dependencies detection...

Would that be useful ?

@shlevy shlevy added the backlog label Apr 1, 2018
@shlevy shlevy self-assigned this Apr 1, 2018
@Ekleog
Copy link
Member

@Ekleog Ekleog commented Apr 25, 2018

Related issue (same objective, different way to do it): #2107

@veprbl
Copy link
Member

@veprbl veprbl commented Jan 19, 2019

I was previously mentioning that "spack" supports some relocation. Today I found out that Homebrew has some limited support for relocation too. It seems like it works only in cases when it can be done without rewriting:
https://github.com/Homebrew/brew/blob/60428d4dcb5a0b31fd48f69356282f4855b15404/Library/Homebrew/dev-cmd/bottle.rb#L357-L370
Makes me wonder how often it is the case.

@matthewbauer
Copy link
Member

@matthewbauer matthewbauer commented Feb 27, 2019

I have a PR for a statically built Nix here: NixOS/nixpkgs#56281

It requires the user namespace feature to be available, like with nix run. But it makes it easier to do this in embedded environments.

@tomberek
Copy link
Contributor

@tomberek tomberek commented Aug 23, 2019

@timjrd Tested and have used nixrewrite several times. I think it could be sped up, but functions well.

@adrian-gierakowski
Copy link

@adrian-gierakowski adrian-gierakowski commented May 8, 2020

I wonder if this could help with: #2925

@nixos-discourse
Copy link

@nixos-discourse nixos-discourse commented Jul 17, 2020

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/idea-for-very-minimally-changing-nix-cache-nixos-org-to-allow-custom-store-paths-without-rebuilding-the-world/8153/4

@stale
Copy link

@stale stale bot commented Feb 13, 2021

I marked this as stale due to inactivity. → More info

@stale stale bot added the stale label Feb 13, 2021
@veprbl
Copy link
Member

@veprbl veprbl commented Feb 13, 2021

Still important to me

@stale stale bot removed the stale label Feb 13, 2021
@stale
Copy link

@stale stale bot commented Aug 13, 2021

I marked this as stale due to inactivity. → More info

@stale stale bot added the stale label Aug 13, 2021
@veprbl
Copy link
Member

@veprbl veprbl commented Aug 13, 2021

Still important to me

@stale stale bot removed the stale label Aug 13, 2021
@Ericson2314
Copy link
Member

@Ericson2314 Ericson2314 commented Aug 14, 2021

We have more path rewriting functionality from the CA stuff, but we should keep on the heat on distros to allow unprivileged users to sandbox processes too I think!

@veprbl
Copy link
Member

@veprbl veprbl commented Aug 14, 2021

The unprivileged user namespaces are enabled in RHEL8 by default, so many sites should get the feature in the following 5 years. The sandboxing is not a complete solution, though. It only works when already running in the sandbox, but not when things are to be shared with other systems (e.g. crontab, job submission, other users).

@nomeata
Copy link
Contributor

@nomeata nomeata commented Feb 16, 2022

It doesn’t rewrite nix paths, but still worth linking from here: https://github.com/DavHau/nix-portable simulate the /nix/store directory, and may work in some cases.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

17 participants