run via slurm: permission denied

[biocyberman]

I installed enroot version enroot-hardened+caps_2.0.1-1 on both frontend node and compute node n1.

I opened an interactive shell:

ADuser@frontend:~/salloc --nodelist=n1 --ntasks=1 -c 2 --gres=gpu:1 srun --pty bash -l

and then ran this command:

enroot list

And got this error:

mkdir: cannot create directory ‘/run/user/140196’: Permission denied

If I login with a local user to n1 and run enroot list it works. It also works if I run the same command on frontend node with ADuser: ADuser@frontend:~/ enroot list.

[3XX0]

The enroot defaults rely on XDG. In this case, this is related to XDG_RUNTIME_DIR.

My guess is that this directory doesn't exist because on Linux, it is usually created by systemd-logind which requires PAM and a login shell. SLURM (UsePAM) with systemd doesn't work really well so it's probably what's happening here.

To fix it you can use ENROOT_RUNTIME_PATH either in the environment or /etc/enroot/enroot.conf to have a directory where users can write to, alternatively, you can create the XDG_RUNTIME_DIR in a prolog.

Somewhat related since you're using SLURM, you might want to look at https://github.com/NVIDIA/pyxis

[biocyberman]

@3XX0 Thanks for the pointers. I set ENROOT_RUNTIME_PATH in /etc/enroot/enroot.conf to /tmp/enroot/${UID} and it works. For future readers, you may need to run chmod 1777 /tmp/enroot so everyone can write to it.