Possible Workarounds for Inability to Mount as non-root under Linux

[stellarpower]

System information

Distribution Name | Ubuntu Server
Distribution Version | 20.04
Linux Kernel | 5.4.0-1012-raspi #12-Ubuntu
Architecture | aarch64
ZFS Version | 0.8.3-1ubuntu12 (dkms)
SPL Version | 0.8.3-1ubuntu12 (dkms)

Describe the problem you're observing

It seems to be a well-known problem that mounting datasets as non-root users on linux is not possible. I have seen a variety of possible options discussed online, for example, here and here.

After encountering the issue earlier in a manner that is preventing me from completing my system setup, a couple of possible workarounds has come to mind, a hackier version of one of which I have successfully implemented to an extent in Crystal (daemon running as root). If, as it seems, the kernel limitation that prevents the current OpenZFS implementation on linux from being able to mount these datasets as non-root looks like it won't be changing any time soon, I am opening this issue as a potential way to discuss any workaround options that may be feasible.

I feel this specific missing feature is a significant factor limiting my being able to use zfs on a number of my linux systems, which is a great shame as whilst not perfect, it is a huge step up from e.g. ext4. It appears to me that perhaps previous discussions naturally concentrated on the more logical solution, fixing the core of the problem at the lower level with the issues with the mount syscall in the global context. If making any required changes with this will not be possible with limitations in the linux kernel that cannot be overcome for a significant amount of time, then I am opening this issue here to open up the discussion if any less elegant but workable solutions might be considered, implemented, and integrated in some way in order to provide a 1) satisfactory, 2) secure enough and 3) workable solution to the problem, where (3) is in the sense that even if not ideal, the solution will function correctly on a linux system, and (2) implies that the fix will not compromise security in any way above the normal levels of risk - for example, if such a fix provides a new vector that could be used to compromise security (e.g. listening as root on a socket) but this vector can and should be locked off under a normal setup (appropriate permission on this socket to prevent misuse), then we could consider (2) to have been met.

Even if a workaround feature is not ideal, I believe it would be worth considering in some level of detail, as for the reasons given below in the motivation, it seems ZFS is coming towards linux more often these days, and has been adopted quite fondly within the BSD community, in any case, quite a way beyond the initial illumos/Solaris base for which it was designed.

Motivations:

Please note most of these are somewhat personal, I will add to them over time if others would like to put forward their own use cases

• ZFS is far superior in many respects to (e.g.) ext4, and I personally would like to move to root on zfs for all of my systems. Without the ability to respect mount permissions on the datasets, this would not be possible on a multi-user system, at least, not without significant compromise in the ability of unprivileged users to perform any actions with their own datasets, which is a large motivating reason for the use of ZFS in the first place.
• Canonical seems to be exploring the option of root on ZFS as an option for mainline Ubuntu going forwards, and this issue would potentially put a stop in any future integrations if not solved in some manner.
• rootless docker: issues with the overlayfs driver. ZFS seems to work out of the box whilst the default overlayfs driver is more complicated as non-root. Docker is a very popular tool in the world of containerisation and rootless docker may well gain popularity as it develops over time to fix some small features that are lacking.

Given the prevalence of docker and Ubuntu currently, and that docker's main reference support for linux lies with Ubuntu, I reckon that if both of these platforms are looking to make ZFS support more standard and tightly integrated in the future, then that would be a good indication that a number of linux projects may seek to utilise ZFS as standard.

• FUSE support: I have seen many articles online reference the FUSE implementation of zfs, however, I believe the current opinion is that it hasn't been maintained for a long time and was far from complete when it was. Without a FUSE option, it will be impossible for any unprivileged userspaced actions involving ZFS datasets requiring any of the currently disallowed operations.

Potential Workarounds

• Utilise a helper binary

Refactor the operations requiring root privileges into a distinct helper binary
Make use of an existing infrastructure for running of this binary as root,

• Any internal commands that require superuser privileges can call out of the binary and become root to execute the required operation via a helper binary before returning back to unprivileged execution
• So for a mounting operation, we would check the user's privileges on the dataset, bailing here if the user lacks permissions, otherwise calling e.g. sudo zfs-helper mount Pool/dataset options
  Options include
• Use the setuid/seteuid bit on this binary
  • in this case, the helper binary would have to do a rigorous permissions check again on the action being requested; the standard zfs command could provide a more user-friendly error message with the assumption that the helper is not to be called directly, and so will also have to perform the same checks, however as the helper will run as root, it will be necessary to ensure the operation being requested is valid before executing it lest it be co-erced into an illicit operation
• Use the sudo system
  • Use the sudo system to become root in order to execute the helper binary. This would require much the same as above, however allow a finer grain of control at the OS-level for disallowing some users from ever performing these operations. The correct setup of the sudoers file would come down to the end user-administrator, however including recommended setup for minimal/everyday privileges would be helpful. It should be noted that whilst prevalent not all distributions come with sudo installed at all.
• Other options
  • I must admit I am aware of, but don't know anything about, POSIX capabilities; same goes for what options if any SELinux and FACLs might have to do with the matter

• Run a mounting deamon

  Place all the functionality that is required for mounting/unmounting (and any other privileged operations) of the datasets into a separate application that runs by default as a deamon, as root.
• The zfs program will perform basic validation of commands, ensures that the calling user passes the delegated permissions, and when any operation requires something that currently the user has permissions to perform, but cannot, due to the limits in the kernel, it passes a message to the daemon, over a socket with (global) write access.
• The daemon will once again verify that the user has the correct permissions for the operation (as it might be required to listen on a socket to which any user could connect, however the first verification would potentially provide a more helpful error message), before performing the operation, inherently as root.
• systemd or equivalent may be used to start this deamon correctly at boot, before the mount targets are met but optionally after importing the required pools.
• The socket could be made globally writable, and the daemon can obtain the effective uid of the calling process to ensure that the requested operation is allowed for said user.

I open the floor up for any comments on the subject and will edit the issue text to add any options as they arise.

[clinta]

To add to this discussion, there are existing tools designed to allow non-privileged users to mount removable devices, including pmount and udisks. I'm not sure if either of those could be utilized for mounting datasets, since they're both designed for mounting devices in /media, but it might be an option.

[ivzhh]

Personally I prefer the SUID solution. Basically all the implementation requires a root/CAP_SYS_ADMIN binary (running or with suid bit). That binary must examine permission and input options carefully to prevent privilege-escalation attack.

Podman and other daemon-less, rootless approaches try to avoid a daemon. Especially if the daemon is stateful, a hotfix will be problem (e.g. dbus-daemon).

I am trying to implement it as a standalone helper binary, instead of patching mount.zfs. If it works, I can try to use it for the zfs graph driver in containers project, which currently uses syscall mount.

[stellarpower]

This is the architecture being used on ZSys, which seems to be a relatively major concentration of effort for Ubuntu's purposes regarding enhanced ZFS support. I believe the architecture was developed in communication with the OpenZFS Dev team, and might be worth looking at . I agree that the SETUID option would be quite easy to achieve, and in fact it may be possible mostly to move parts of the existing codebase and add some boilerplate in order to separate it, but equally if a more client-server model is what would be best adopted by other projects, then maybe this would be easier forming a part of the core interface, or perhaps wrapping around the C library as an alternative to facilitate interaction at a programmatic level rather than calling out to the CLI utilities. Regarding that, I believe Joyent was wrapping some core Solaris libraries in node.js to help with some of their tools.

Also, AFAIK, but I may be wrong, things like udisks require FUSE support, and it seems the zfs-fuse package is long unmaintained and has been deprecated in many places I have read, however maybe that would be a more appropriate model, or again, some modest changes that would bridge the gap a little and make maintenance of a FUSE interface easier.

[stellarpower]

@ivzhh Do you have a repo for your helper binary? Sounds quite similar to something I was trying to do for the rootless docker setup. I'm lacking time right now but would be happy to have a look at it if I did get round to it, especially as this was preventing me from using the rootless setup after quite an investment into it for my server.

[ivzhh]

@stellarpower Not yet. I am extending go-libzfs and still work-in-progress. A basic version will be ready in a week or two.

Glad to hear you are working towards the same direction.

[ivzhh]

@stellarpower Zsys is a cool project and it led me to use go-libzfs. According to Zsys,

Package authorizer deals client authorization based on a definite set of polkit actions.
The client uid and pid are obtained via the unix socket (SO_PEERCRED) information,
that are attached to the grpc request by the server.

I really like the solution too. In addition, my solution is go-based and go cannot do syscall.Setuid() due to the threading model. I need to use libcap/psx for Setuid(). So I have many reasons not to use SUID path.

However, the reason I tried to go through SUID is daemonless. For people like podman, rootless and daemonless are two important values. So that's the main reason why I chose SUID.

[clinta]

I know you're asking for a workaround, but I've been digging more into ZFS internals and I wonder if the right way to fix this would be to expose another ZFS ioctl for mounting move calling the mount syscall into the zfs module. Then delegation can all be verified the same way all other ioctl calls are.

[khenriks]

I came across this thread while trying to mount as non-root, but it didn't seem like there was any ready-to-use solution available. After futzing with it for way too long, I came up with this solution for doing what I needed.

This uses the CAP_SYS_ADMIN capability to grant the ability to mount, and is only capable of mounting using the mountpoint property embedded in the dataset (this is a security feature). It restricts usage to a single user as a security mitigation, but could easily be extended to accommodate additional users.

Unfortunately, it does not perform any authorization checks to determine if the user is permitted to mount the dataset (via zfs allow). As far as I can tell, the ability to do this check is not exposed anywhere. Restricting to a single user is the mitigation I settled on for this, but a new ioctl to expose the authorization check to userspace would be extremely welcome and make this more useful.

[stellarpower]

Thakns for this! Zero chance I can look at this thread properlky for the next 6 months, but, I remember doing something in a similar manner years ago for dropbear:

stellarpower/dropbear@d17dedf

Thought I'd include just in case you were interested in expanding.

[udf2457]

@khenriks Could you tell me a little more about how you actually use this ?

i.e. the typical example is zfs send a/b@c | ssh remote zfs receive a/b, how does your binary tie into this ? As far as I can tell you can't tell zfs receive to call a "hook" ?

[stellarpower]

This is probably going to sound stupid, but I'm assuming namespaces don't allow us any leverage here. My understanding is that this would be limited, as if the dataset could be mounted within the non-root dataset, it still can't be viewed outsidde of that namespace. But, maybe someone knows the topic in a lot more intricate detail and might be aware of a hack, or way this can at least be used to some effect. E.g., if we're talking setuid workarounds, helper binaries, etc., then would it be less harmful to run some sort of server process in the namespace that can then feed bak to the root namespace as a normal user.

[ghost]

I've written a small zfs-mount-helper program in C which is similar to the one @khenriks provided, however it checks if the user has the permission to mount the target dataset according to zfs allow.

It does so by using zfs_get_fsacl() to grab the list of permission entries, iterates over it, checks if the user/group ID executing the mount helper matches the UID/GID of an allow entry (or the permission is granted to everyone) and then checks if the permission list for the UID/GID contains a mount entry.

An alternative solution, as @clinta mentioned, would be to expose the verification procedure with a new ioctl. I think this is preferrable to this solution -- however, this one doesn't need changes on the kernel-level.

!!NOTENOTENOTE!!
I hacked this tool togeher in 2-3 hours while performing some minor side-tasks. I tested it on a simple pool the user does not have mount access to, containing two sub-datasets: One datasets grants the mount permission based on the UID, one based on everyone. (i.e., the case granting the mount permission based on the group ID hasn't even been tested yet). I still have to check the logic for soundness, write tests and teach zfs a way to use this mount-helper.
So please, don't use it for anything even remotely productive -- I just publish it for reference, and to show that there's some progress on this.
!!NOTENOTENOTE!!

I usually use a self-hosted gitea instance to publish my code and would link to it here but I don't have access to it because I don't carry my MFA token with me right now, so I just copy-paste this experimental code here.

// Optionally add -O0 -g3 for debugging.
//
// gcc $(pkg-config --cflags libzfs) zfs-mount-helper.c -o zfs-mount-helper $(pkg-config --libs libzfs)
// setcap cap_sys_admin+ep zfs-mount-helper
//
#define _LARGEFILE64_SOURCE
#include <libzfs.h>
#include <zfs_deleg.h>
#include <stdlib.h>
#include <stdio.h>
#include <stdbool.h>

// __attribute__((__cleanup__(...))) helper
void __cleanup_libzfs_handle_p(libzfs_handle_t **libzfs) {
	if(*libzfs) libzfs_fini(*libzfs);
}
void __cleanup_zfs_handle_p(zfs_handle_t **zfs) {
	if(*zfs) zfs_close(*zfs);
}


// Checks if a user can mount the given zfs dataset
// returns 1 if:
// - given UID can be found in the ACL and has the `mount` permission set.
// - given GID can be found in the ACL and has the `mount` permission set.
// - the ACL contains the `mount` permission for `everyone`.
int check_can_mount(zfs_handle_t *zfs, uid_t uid, gid_t gid) {
	libzfs_handle_t *libzfs = zfs_get_handle(zfs);

	nvlist_t *fsacl_nvl = NULL;
	int rc = zfs_get_fsacl(zfs, &fsacl_nvl);
	if(rc) {
		fprintf(stderr, "zfs_get_fsacl(): Error %d --  %s\n", rc, libzfs_error_description(libzfs));
		return -1;
	}

	// We only query the permission for one dataset, so we don't use a
	// while() loop here, but only call nvlist_next_nvpair once.
	//
	// Note that, if the dataset does not contain any kind of allowed permissions, nvlist_next_nvpair() returns
	// NULL -- NULL is also returned if an error occurs, so we have to distinguish error and non-existing permissions
	// here.
	nvpair_t *fsacl_nvp = NULL;
	if((fsacl_nvp = nvlist_next_nvpair(fsacl_nvl, fsacl_nvp)) == NULL) {
		if(libzfs_errno(libzfs) != 0) {
			fprintf(stderr, "nvlist_next_nvpair(): %s\n", libzfs_error_description(libzfs));
			return -1;
		}
		goto nope;
	}

	// Small sanity check that we got the ACL info of the right dataset &
	// it has the right type
	const char *actual_dataset = nvpair_name(fsacl_nvp);
	const char *dataset_name = zfs_get_name(zfs);
	if(strncmp(dataset_name, actual_dataset, strlen(dataset_name))) {
		fprintf(stderr, "Requested ACL for dataset %s, but zfs_get_fsacl() returned ACL for dataset %s\n", dataset_name, actual_dataset);
		return -1;
	}
	data_type_t fsacl_nvp_type = nvpair_type(fsacl_nvp);
	if(fsacl_nvp_type != DATA_TYPE_NVLIST) {
		fprintf(stderr, "Returned ACL entry type is not DATA_TYPE_NVLIST (%d), but %d\n", DATA_TYPE_NVLIST, fsacl_nvp_type);
		return -1;
	}


	// Grab the nvlist that contains the list of allowed permissions.
	// This nvlist contains an nvpair with name '<ZFS_DELEGATION_TYPE:char><ZFS_LOCALITY:char>$<NOTHING_OR_UID_OR_GID>'
	// TODO: at least that's the relevant cases for now, I still have to dig deeper to check what else might be written in there
	//
	// The value of this nvpair again is an nvlist containing the list of allowed permissions. This list contains nvpairs with
	// a name corresponding to the permission ("mount", "unmount", "create", "destroy", ...) and a value data type of
	// DATA_TYPE_BOOLEAN. Note that a DATA_TYPE_BOOLEAN indicates a value of B_TRUE just by existing as an nvpair, so we don't
	// have to grab the boolean value using `nvpair_value_boolean_value()`.
	//
	// So, the logic we will perform is:
	// - Iterate over the list of entries
	// - If the permission is granted to a user, check if the UID matches the one executing this program.
	// - If the permission is granted to a group, check if the GID matches the one executing this program.
	// - If the permission is granted to everyone, just continue.
	// - Check if the granted permission list allows the `mount` permission.
	// - return 1 if it does, fall-through (0) if not.
	nvlist_t *permnvl = NULL;
	if(nvpair_value_nvlist(fsacl_nvp, &permnvl) != 0) {
		fprintf(stderr, "Could not get permission nvlist for dataset %s: nvpair_value_nvlist(): %s\n", dataset_name, libzfs_error_description(libzfs));
		return -1;
	}

	nvpair_t *permnvp = NULL;
	while((permnvp = nvlist_next_nvpair(permnvl, permnvp)) != NULL) {
		data_type_t permtype = nvpair_type(permnvp);
		if(permtype != DATA_TYPE_NVLIST) {
			fprintf(stderr, "Returned permission entry type is not DATA_TYPE_NVLIST (%d), but %d\n", DATA_TYPE_NVLIST, permtype);
			return -1;
		}


		const char *permname = nvpair_name(permnvp);
		if(permname[2] != '$') {
			fprintf(stderr, "Returned permission entry has bogus name -- its third position MUST be '$'. Value: %s\n", permname);
			return -1;
		}
		// ignored for now?
		// char deleg_locality = permname[1];
		//if(deleg_locality != ZFS_DELEG_LOCAL)
		//	continue;

		// Check if the delegation type is making this permission
		// relevant for this user/group
		char deleg_type = permname[0];
		const char *id = &permname[3];
		uid_t permuid;
		uid_t permgid;

		bool perm_match = false;
		switch(deleg_type) {
			case ZFS_DELEG_USER:
				permuid = atoi(id);
				if(uid == permuid) perm_match = true;
				break;
			case ZFS_DELEG_GROUP:
				permgid = atoi(id);
				if(gid == permgid) perm_match = true;
				break;
			case ZFS_DELEG_EVERYONE:
				perm_match = true;
				break;
			default:
				// Not supported, ignored.
				continue;
		}

		if(perm_match == false)
			continue;

		// Check if the mount permission is part of the list.
		nvlist_t *allownvl;
		if(nvpair_value_nvlist(permnvp, &allownvl) != 0) {
			fprintf(stderr, "Could not extract list of allowed permissions from perm %s\n", permname);
			return -1;
		}

		nvpair_t *allownvp;
		while((allownvp = nvlist_next_nvpair(allownvl, allownvp)) != NULL) {
			const char *allowname = nvpair_name(allownvp);
			if(!strcmp("mount", allowname)) {
				data_type_t allowtype = nvpair_type(allownvp);
				if(allowtype != DATA_TYPE_BOOLEAN) {
					fprintf(stderr, "Found 'mount' permission, but it's not of DATA_TYPE_BOOLEAN (%d), but %d\n", DATA_TYPE_BOOLEAN, allowtype);
					return -1;
				}

				fprintf(stdout, "User with UID/GID %d/%d is allowed to mount ZFS dataset %s!\n", uid, gid, dataset_name);
				return 1;
			}
			else continue;
		}
	}

nope:
	fprintf(stdout, "User with UID/GID %d/%d is NOT allowed to mount ZFS dataset %s!\n", uid, gid, dataset_name);
	return 0;
}

int main(int argc, char *argv[]) {

	if(argc < 2 || argc > 3) {
		fprintf(stderr, "Usage: %s ZFS_DATASET_NAME\n", argv[0]);
		return EXIT_FAILURE;
	}

	libzfs_handle_t *libzfs __attribute__((__cleanup__(__cleanup_libzfs_handle_p))) = libzfs_init();
	if(!libzfs) {
		fprintf(stderr, "libzfs_init(): %s\n", libzfs_error_init(errno));
		return EXIT_FAILURE;
	};

	zfs_handle_t *zfs __attribute__((__cleanup__(__cleanup_zfs_handle_p))) = zfs_open(libzfs, argv[1], ZFS_TYPE_FILESYSTEM);
	if(!zfs) {
		fprintf(stderr, "zfs_open(): %s\n", libzfs_error_description(libzfs));
		return EXIT_FAILURE;
	}

	if(!check_can_mount(zfs, getuid(), getgid())) {
		return EXIT_FAILURE;
	}

	fprintf(stdout, "Attempting to mount dataset %s...", zfs_get_name(zfs));
	if (zfs_mount(zfs, "", 0)) {
		fprintf(stdout,"FAILED!\n");
		fprintf(stderr, "zfs_mount() failed: %s\n",
				libzfs_error_description(libzfs));
		return EXIT_FAILURE;
	}
	fprintf(stdout,"Success!\n");

	return EXIT_SUCCESS;
}

[ghost]

Quick update after further investing some time: zfs_get_fsacl() can return more than one nvlist, I got that completely wrong. Thus the "sanity check" is completely wrong too.

I'll fix these -- among others I still have to locate :-) -- issues, add the ability to unmount & write some tests over the next few days.
Just reach out if you have any questions or wishes.

[comphuter]

Note: @mbiberhofer is my work account and will be suspended soon. Sorry for the confusion.

Fixed the previously noted issues. You can find the current up-to-date code at https://git.skyforge.at/comphuter/zfs-mount-helper (install not supported currently. -- just so you know, the binary needs the CAP_SYS_ADMIN capability or SUID to work.)

Some notes:

• zfs share doesn't work. The zfs command needs adoption to use this helper. I already have some hack-ish code flying around that implements this, but I think it should be discussd if we really should add support for such a hotfix to the zfs command. Upstream won't merge code related to this anyway I guess.
• -u switches the helper to unmount mode, unmounting the target dataset instead of mounting it.
• I think we should tackle the issue at the core: We can't mount anything in the root mount namespace without being privileged. This solution should be considered merely a hotfix.
• The helper could of course also be adapted for zfs share/zfs unshare or any other operation. Right now it only supports (un)mounting.
• I published the code under the MIT License

Any ideas on how to tackle the issue at a more appropriate level? I'm in a bit of a loss here: I really don't know how we can enable unprivileged linux users to mount zfs datasets without a privileged helper.

I'm willing to invest even more spare time into solving this issue, I'd like to "do it right" though. Feedback is appreciated!

[adrolter]

@comphuter Thanks for this! I managed to compile it and get it to work from a shell, but is this supposed to integrate with zfs mount somehow?

[comphuter]

EDIT: First things first. Sorry I didn't make that clear -- no, it currently can't integrate with zfs mount. zfs-mount-helper is just the proof-of-concept shell tool. :-)

The most straight-forward way to integrate the hotfix into zfs mount seems like:

1. integrate the zfs acl permission check into mount.zfs (cmd/mount_zfs.c, between #ifdef __linux__ guards) &install the binary with CAP_SYS_ADMIN (on linux).
2. setting the environment variable ZFS_MOUNT_HELPER=1 makes zfs mount use /sbin/mount instead of the mount(2) syscall, in turn calling mount.zfs
3. The ZFS_MOUNT_HELPER=1 variable also makes zfs unmount use /sbin/umount. We need to implement a umount.zfs binary for performing the acl-check and unmounting the filesystem.

I already implemented the mount.zfs integration and tested the steps (1,2) on my development environmet successfully -- I still have some fixes, modifications & cleanups on TODO though. I'll get that done, then implement unmount.zfs.
I think I'll get both done tomorrow/ the day after tomorrow.

It is hack-ish and a hotfix, but I think for most users it is a sufficient solution until we find a better one.

[adrolter]

A-ha, that's what you were driving at by "install not supported currently"...I think I inferred too much from your comment about the helper being able to support zfs share in the future, but only supporting (un)mounting currently.

Thank you so much for investing time and effort into this! It's greatly needed and appreciated.