Explore the feasibility of using fuse inside running jupyter sessions

[olevski]

Something that may be beneficial/useful is this project:

• https://community.arm.com/arm-research/b/articles/posts/a-smarter-device-manager-for-kubernetes-on-the-edge
• https://gitlab.com/arm-research/smarter/smarter-device-manager

In theory this could allow users to access and use fuse mounts without elevated permissions. We should check whether using this in our cluster would enable us to do fuse mounts without modifying our pod security policies and escalating permissions.

[Panaetius]

Timeboxed to 3 days

[olevski]

See here: https://github.com/SwissDataScienceCenter/renku-notebooks/tree/use-fuse-in-containres-without-root/manifests

The smarter device manager works. But to make this possible I had to increase permissions on the pod:

• Add SYS_ADMIN in the capabilities
• Turn off AppArmor for the container where the mounting happens

This repo https://github.com/samos123/gke-gcs-fuse-unprivileged shows how you can create and load a custom AppArmor profile so that it is not as bad as fully turning off AppArmor. But I am not certain what the implications of this are or how good the suggested custom AppArmor profile is. I think there is a risk here that if we mess something up we could allow a user to "spy" or impersonate other users on the same node.

It could be interesting though to see if a sidecar could be using the above setup. The sidecar could mount inside a volume mounted on both the sidecar and the user session. Then the sidecar can expose a rpc server to perform the mounts and nothing else and the user will not be able to have the same permissions as the sidecar but they can get anything mounted. I am not sure what permissions/features are required to make the sidecar successfully mount fuse stuff in a PV shared with the main container.

[olevski]

Trying to do the fuse mount in a sidecar container and then use them in another (i.e. main) container with a lot less privileges.

Attempt 1:

• put the fuse mount inside a persistent volume
• mount volume to both containers
• use HostToContainer mountPropagation on the volume mounts in both containers
• DOES NOT WORK, the main container cannot access / see anything that is fuse mounted
• Bidirectional mountPropagation would work here but this requires that the container to which it is assigned is running in privileged mode which is exactly what we are trying to avoid

Attempt 2:

• mentioned by FUSE volumes kubernetes/kubernetes#7890 (comment)
• put the fuse mount in a hostPath volume on both containers
• it does not work because the host path volume/location needs root permissions if someone wants to write in it and it can only be adjusted by running an init container with root credentials and running chmod
• DOES NOT WORK

Attempt 3:

• mentioned by FUSE volumes kubernetes/kubernetes#7890 (comment)
• however the comment above says that this can cause the pod to get stuck when you try to delete it
• the idea is that instead of a hostpath you use an emptyDir volume and place the fuse mount there
• DOES NOT WORK

[olevski]

Overall conclusion: lets just stick to using a csi driver for this and restart the pod when we need to mount something. Mounting can also provide only so many benefits because the IO performance when things are mounted will never be that good anyhow.

[Panaetius]

closing this based on the conclusion