How to set up Neurodesk and HPC on a cluster

[bkossows]

Hi everyone,

I’ve been asked by our cluster administrator to submit a request for a solution where JupyterHub could function as a terminal for multiple users, while also allowing unused resources to be utilized for heavy computational tasks (e.g., parallel processing with recon-all).

The problem is, I’m not entirely sure how to describe such a setup or what tools/configurations could make this possible. Essentially, we’re looking for a way to:

• Let users work interactively through JupyterHub. Most users do not need more than 4CPUs to run fsl commands and check the results.
• Use idle resources for demanding jobs when available.

If anyone has experience with similar setups or can suggest how to approach this, I’d really appreciate your advice. Any tips on integrating this with a job scheduler like SLURM or Kubernetes would also be super helpful.

Kind regards,
Bartosz Kossowski

[stebo85]

Dear @bkossows

There are two solutions that come to my mind:

1. Run a standard SLURM cluster and install OpenOndemand (https://openondemand.org/). OpenOndemand will provide you with Jupyter and full desktop support for interactive computing. Users can then submit longer running jobs to the underlying slurm cluster. Then install Neurodesk into this setup. This is the setup most universities have and what we run as well - if you want I can show you this setup in a zoom call: https://outlook.office.com/bookwithme/user/5b38d212870546ce88633f7b65d7a2d2@uq.edu.au?anonymous&ep=pcard

2. A Kubernetes cluster: This is something we are currently developing, but it still needs a lot of Kubernetes expertise to run. We run Jupyterlab with Neurodesk on Kubernetes and we have a notebook submission system that's powered by Kubeflow under the hood. Pipeline creation is managed by Elyra.

[bkossows]

Thank you for your response. I will pass it on to the decision-makers, and I would gladly take the opportunity to meet if they are interested.

[bkossows]

Dear Steffen,
Thank you for your recommendation. So far, we have implemented a Jupyter server using Kubernetes and the Neurodesk Dockerfile (link
) as the main image. We are very happy to start using the platform; however, I have noticed a few potential bottlenecks:

• Loading new containers often takes quite a while. It seems that the CVMFS cache is only 5 GB by default and user-specific ($HOME/cvmfs_cache). Could this be the reason? How would you recommend configuring the CVMFS cache for multiple users on the Jupyter server? Is it possible to set up a shared cache, or would it be better to run a local repository instead?
• We are also planning to start working on a SLURM cluster with Open OnDemand. I think our IT expert still needs to get a better understanding of the setup. Would it be possible to schedule a demo session with You and @p-andrzejewski-fj?

Kind regards
Bartosz Kossowski

[stebo85]

Dear @bkossows,

great to hear you got it working.

Regarding CVMFS there are a few options we recommend:

1. you could run a local squid proxy that all your connections go through and that caches every tool
2. you could run your own stratum 1 server
3. you could run CVMFS as a kubernetes pod and have a shared cache between all user-pods (this is the setup we use for our Kubernetes deployments. I can also share the manifests)

Regarding OpenOndemand: Yes, I can show you how we deploy that. I am currently moving between continents, so a meeting in November would be ideal. Would that work?

[bkossows]

Thanks for the quick reply. As for now please send us the manifests for the third option. We will try to optimize the current system before we move to the OpenOnDemand.
Cheers

[stebo85]

Dear @bkossows,

Here is how you could achieve a shared cvmfs setup between pods:

# #========================================#
# # CVMFS
# #========================================#
 
helm install cvmfs-csi oci://registry.cern.ch/kubernetes/charts/cvmfs-csi --version 2.5.0 -n jupyterhub
 
# Create CVMFS pvc
kubectl create -n jupyterhub -f templates/volume-storageclass-pvc.yaml
 
# Apply Neurodesk CVMFS ConfigMaps
kubectl create -f templates/cvmfs-csi-config-d.yaml --dry-run=client -o yaml | kubectl apply -f -
kubectl create -f templates/cvmfs-csi-default-local.yaml --dry-run=client -o yaml | kubectl apply -f -

kubectl -n jupyterhub get pods -o wide
kubectl delete pod -n jupyterhub cvmfs-csi-controllerplugin-*
kubectl delete pod -n jupyterhub cvmfs-csi-nodeplugin-*
kubectl -n jupyterhub get pods -o wide
 
# echo
# echo "Remember to ..."
# # echo "kubectl edit configmap cvmfs-csi-config-d -n jupyterhub"
# # echo "kubectl edit configmap cvmfs-csi-default-local -n jupyterhub"
# # echo "And add the details in ../cloud/cvmfs-csi-config-d.yaml and cvmfs-csi-default-local.yaml"
# echo "delete the cvmfs nodeplugin pod to apply changes"
# echo "To test/troubeshoot ..."
# echo "Shell into the automount container in the cvmfs-cvmfs-csi-nodeplugin-*** pod"
# echo "mount -t cvmfs neurodesk.ardc.edu.au /mnt -v"
# echo "After testing ... umount /mnt "

volume-storageclass-pvc.yaml
cvmfs-csi-config-d.yaml
cvmfs-csi-default-local.yaml

I hope these instructions still work - they are from an older version of our setup before we moved to argo as a deployment

[stebo85]

@bkossows - just wondering where you are at with this? Would you still like to meet and chat about this?

[bkossows]

Dear Steffen,

We are still considering Neurodesk as our default platform. Unfortunately, our IT administration is currently undergoing reorganisation and cannot help me much further with troubleshooting the environment.

Overall, Neurodesk with JupyterLab and DockerSpawner works quite reliably. The only problem I haven't been able to solve is that JupyterLab (and Guacamole) becomes very unresponsive even under moderate computational load.

Interestingly, this happens on a per-user basis. If I run fMRIPrep on my account, my interface becomes very slow, while other users on the same machine can work normally. The resources are shared (32 CPUs, 120 GB RAM), and fMRIPrep itself runs at the expected speed. We also see no obvious CPU throttling, memory pressure, or I/O saturation (according to my ai-guided tests). The same behaviour can be reproduced with multiple CPU-bound Python processes, without fMRIPrep or Apptainer.

Have you encountered similar behaviour? I would appreciate any hints, especially regarding Jupyter/DockerSpawner settings that I could pass to our IT administrator.

Best,
Bartosz

[stebo85]

Dear @bkossows,

The per-user behavior makes me suspect CPU contention inside the affected user’s container.

JupyterLab, Tomcat, Guacamole, and the user’s computation all run in the same Neurodesktop container. They therefore share the same Docker CPU allocation. If fMRIPrep or several Python processes consume that allocation, JupyterLab and Guacamole may receive too little CPU time. Other users remain responsive because they run in separate containers.

This can happen even when the host has free resources and the computation runs at the expected speed. A Docker CPU quota applies to the whole container. Worker auto-detection can also make the problem worse. For example, a container may have a four-CPU quota while nproc, Python, or OpenMP sees all 32 host CPUs and starts too many workers.

Could your administrator run these commands for the affected container, first while idle and then while the interface is slow?

C=<affected-user-container>

docker inspect "$C" --format \
'NanoCpus={{.HostConfig.NanoCpus}} CpuQuota={{.HostConfig.CpuQuota}} CpuPeriod={{.HostConfig.CpuPeriod}} Cpuset={{.HostConfig.CpusetCpus}} CpuShares={{.HostConfig.CpuShares}} Memory={{.HostConfig.Memory}}'

docker exec "$C" sh -lc '
printf "nproc: "; nproc
if [ -f /sys/fs/cgroup/cpu.max ]; then
    printf "cpu.max: "; cat /sys/fs/cgroup/cpu.max
    echo "cpu.stat:"
    cat /sys/fs/cgroup/cpu.stat
fi
echo "CPU pressure:"
cat /proc/pressure/cpu
'

docker stats --no-stream "$C"

A rapid increase in nr_throttled or throttled_usec during the slowdown would support this explanation.

Two smaller tests would also help:

1. Repeat the CPU test with one or two fewer workers than the container’s CPU allocation. Alternatively, start it with nice -n 10. If JupyterLab becomes responsive while the job continues, the container is suffering from CPU contention.
2. Redirect all output to a file. This rules out large amounts of terminal or notebook output overwhelming the browser connection.

There is also a previous report with similar symptoms: [Docker deployment with nginx causes significant lag](https://github.com/orgs/neurodesk/discussions/355). In that case, direct access was fast, but access through Nginx lagged by 10–15 seconds. Adding the following setting fixed it:

proxy_buffering off;

If you use a reverse proxy, check that it also uses HTTP/1.1 and forwards the WebSocket Upgrade and Connection headers. A good test is to access JupyterHub through an SSH tunnel that bypasses the outer proxy while the CPU workload is running. If the direct connection is responsive but the public connection is slow, the proxy is the likely cause.

For regular use, I would limit heavy jobs to fewer CPUs than the interactive container has available, leaving one or two CPUs for JupyterLab and Guacamole. The more reliable design is to submit heavy work to Slurm and keep the JupyterHub container for interactive work.