Darwin chroot approach?

[wmertens]

As we know, OS X doesn't have bind mounts, but HFS+ does allow hardlinking directories.

So as long as the nix store and all desired impurities are on the same HFS+ filesystem, we could make a directory that hardlinks /nix and the impurities, and run all builds chroot-ed under there.

When I say chroot I'm not talking about the chroot build code (

nix/src/libstore/build.cc

Lines 1750 to 1854 in d98bfcb

	if (useChroot) {
	#if CHROOT_ENABLED
	/* Create a temporary directory in which we set up the chroot
	environment using bind-mounts. We put it in the Nix store
	to ensure that we can create hard-links to non-directory
	inputs in the fake Nix store in the chroot (see below). */
	chrootRootDir = drvPath + ".chroot";
	if (pathExists(chrootRootDir)) deletePath(chrootRootDir);
	/* Clean up the chroot directory automatically. */
	autoDelChroot = std::shared_ptr<AutoDelete>(new AutoDelete(chrootRootDir));
	printMsg(lvlChatty, format("setting up chroot environment in ‘%1%’") % chrootRootDir);
	/* Create a writable /tmp in the chroot. Many builders need
	this. (Of course they should really respect $TMPDIR
	instead.) */
	Path chrootTmpDir = chrootRootDir + "/tmp";
	createDirs(chrootTmpDir);
	chmod_(chrootTmpDir, 01777);
	/* Create a /etc/passwd with entries for the build user and the
	nobody account. The latter is kind of a hack to support
	Samba-in-QEMU. */
	createDirs(chrootRootDir + "/etc");
	writeFile(chrootRootDir + "/etc/passwd",
	(format(
	"nixbld:x:%1%:%2%:Nix build user:/:/noshell\n"
	"nobody:x:65534:65534:Nobody:/:/noshell\n")
	% (buildUser.enabled() ? buildUser.getUID() : getuid())
	% (buildUser.enabled() ? buildUser.getGID() : getgid())).str());
	/* Declare the build user's group so that programs get a consistent
	view of the system (e.g., "id -gn"). */
	writeFile(chrootRootDir + "/etc/group",
	(format("nixbld:!:%1%:\n")
	% (buildUser.enabled() ? buildUser.getGID() : getgid())).str());
	/* Create /etc/hosts with localhost entry. */
	writeFile(chrootRootDir + "/etc/hosts", "127.0.0.1 localhost\n");
	/* Bind-mount a user-configurable set of directories from the
	host file system. */
	PathSet dirs = tokenizeString<StringSet>(settings.get("build-chroot-dirs", string(DEFAULT_CHROOT_DIRS)));
	PathSet dirs2 = tokenizeString<StringSet>(settings.get("build-extra-chroot-dirs", string("")));
	dirs.insert(dirs2.begin(), dirs2.end());
	for (auto & i : dirs) {
	size_t p = i.find('=');
	if (p == string::npos)
	dirsInChroot[i] = i;
	else
	dirsInChroot[string(i, 0, p)] = string(i, p + 1);
	}
	dirsInChroot[tmpDir] = tmpDir;
	/* Make the closure of the inputs available in the chroot,
	rather than the whole Nix store. This prevents any access
	to undeclared dependencies. Directories are bind-mounted,
	while other inputs are hard-linked (since only directories
	can be bind-mounted). !!! As an extra security
	precaution, make the fake Nix store only writable by the
	build user. */
	createDirs(chrootRootDir + settings.nixStore);
	chmod_(chrootRootDir + settings.nixStore, 01777);
	foreach (PathSet::iterator, i, inputPaths) {
	struct stat st;
	if (lstat(i->c_str(), &st))
	throw SysError(format("getting attributes of path ‘%1%’") % *i);
	if (S_ISDIR(st.st_mode))
	dirsInChroot[*i] = *i;
	else {
	Path p = chrootRootDir + *i;
	if (link(i->c_str(), p.c_str()) == -1) {
	/* Hard-linking fails if we exceed the maximum
	link count on a file (e.g. 32000 of ext3),
	which is quite possible after a `nix-store
	--optimise'. */
	if (errno != EMLINK)
	throw SysError(format("linking ‘%1%’ to ‘%2%’") % p % *i);
	StringSink sink;
	dumpPath(*i, sink);
	StringSource source(sink.s);
	restorePath(p, source);
	}
	regularInputPaths.insert(*i);
	}
	}
	/* If we're repairing or checking, it's possible that we're
	rebuilding a path that is in settings.dirsInChroot
	(typically the dependencies of /bin/sh). Throw them
	out. */
	if (buildMode != bmNormal)
	foreach (DerivationOutputs::iterator, i, drv.outputs)
	dirsInChroot.erase(i->second.path);
	#else
	throw Error("chroot builds are not supported on this platform");
	#endif
	}
	else {

) but just plain chroot().

To make a hardlink you need to call link(), there's an example CLI utility hlink at http://stackoverflow.com/a/805001/124416 and the very important counterpart hunlink at http://stackoverflow.com/a/4707231/124416 (if you delete files in hardlinked directories, they're gone everywhere - you need to unlink the hardlink).

I tried creating /nix/var/pure with a hardlinked nix/store and some libraries, and it does work, but I can't make chroot work yet 😅. I need a closure of all libraries I suppose, which I think I have but when I do a chroot it just exits without printing anything.

As to why this is needed, I built bash and it picked up libraries from an old homebrew install 😢.

A completely different approach would be to export root over nfs to localhost and use nfs instead of bind mounts.

More eyeballs appreciated.

[copumpkin]

👍

[copumpkin]

Also, bind mounts are quite possible if we set up osxfuse and bindfs. Probably less of a pain than NFS mounts, at least, but I think the hard link thing should work so with any luck we won't need any of that 😄

[wmertens]

Another approach is fakeroot which also works without root privs or fuse
and is probably faster than bindfs

Wout.
On Oct 3, 2014 8:54 AM, "Daniel Peebles" notifications@github.com wrote:

Also, bind mounts are quite possible if we set up osxfuse and bindfs.
Probably less of a pain than NFS mounts, at least, but I think the hard
link thing should work so with any luck we won't need any of that [image:
😄]

—
Reply to this email directly or view it on GitHub
#361 (comment).

[wmertens]

Err, scratch the above, I meant https://github.com/dex4er/fakechroot not https://alioth.debian.org/projects/fakeroot/ and while the former allows you to make a chroot with external symlinks, it doesn't work on OS X. The latter does work on OS X, but doesn't provide chroot 😢.

So besides hardlinking that leaves only NFS or osxfuse, where the latter is reportedly less reliable + slower.

I got chroot to work by simply hardlinking all of /usr/lib, which is obviously not the general idea... Or is it possible to build a pure stdenv from there?

[copumpkin]

Not sure it counts as pure, but have you seen my recent stuff hooking up the Apple command-line utilities to nix (and I'll push a sensible xcode package this weekend), using my fetchadc function that downloads from Apple it you set your credentials in config.nix?

I'm still pretty new to nix, so don't know how stdenv is made, but it seems like the Apple command-line tools package is likely a good starting point. See NixOS/nixpkgs#4365.

[wmertens]

I'm following it with interest!

[wmertens]

Well - that's interesting... I had to reboot because of the OSX update, and after the reboot, my /nix/store was entirely empty 😱.

Perhaps it was because I violated one of the 6 rules mentioned at http://stackoverflow.com/a/4707231/124416, namely that /nix/store and /nix/var/pure share a common parent. I thought it would just fail to hardlink but it seems that it just hardlinks and then uncovers bugs. Sigh.
Luckily, my /usr/lib hardlink didn't result in an empty /usr/lib.

I propose to never ever use directory hardlinks 😅.

I think I'm going to set up dual boot. OS X is becoming a sad ghost of a real Unix system. Not including ZFS and instead using those stupid directory hardlinks, explicitly removing nullfs, removing ptrace() and not providing a granular permissions system for dtrace(), dropping support for gcc... Might as well use an iPad. Not happy 😠.

So anyway, closing this since we have #317. Looks like it's between bindfs or nfs mounts now.

[copumpkin]

TIL about sandbox-exec:

SANDBOX-EXEC(1) BSD General Commands Manual SANDBOX-EXEC(1)

NAME
sandbox-exec -- execute within a sandbox

SYNOPSIS
sandbox-exec [-f profile-file] [-n profile-name] [-p profile-string] [-D key=value ...] command [arguments ...]

DESCRIPTION
The sandbox-exec command enters a sandbox using a profile specified by the -f, -n, or -p option and executes command with arguments.

 The options are as follows:

 -f profile-file
         Read the profile from the file named profile-file.

 -n profile-name
         Use the pre-defined profile profile-name.

 -p profile-string
         Specify the profile to be used on the command line.

 -D key=value
         Set the profile parameter key to value.

SEE ALSO
sandbox_init(3), sandbox(7), sandboxd(8)

Mac OS X July 29, 2008 Mac OS X

[copumpkin]

Check your /usr/share/sandbox directory to see examples of what kinds of things you can limit. Looks like we have very nice granular controls over network access, syscalls, filesystem access. No need for chroot, bind mounts, or anything of the sort. I'm pretty excited.

Also, some unofficial information at http://dl.packetstormsecurity.net/papers/general/apple-sandbox.pdf.

[copumpkin]

Maybe worth reopening the ticket?

[copumpkin]

Sorry for the spam, but if you want to experiment:

sandbox-exec -n no-write bash will give you a bash shell that can't write anywhere.

sandbox-exec -n no-internet bash gives you one that can't do any networking (ping says operation not permitted)

And if you pass in -f you can restrict it to see only specific filesystem paths.

As an added benefit, none of this requires root/sudo.