Request for Guidance on Overlay Conflict Error in Neurodesk (CentOS 7.9)

[B8017913]

Dear Experts,

Our lab is running CentOS Linux release 7.9.2009 (Core).

Following the official instructions (Option 2), I successfully installed the neurodesktop image and made it available to all lab members through our network.

However, when any user tries to launch a specific application inside the container, we consistently hit the following error:

(base) jovyan@neurodesktop-2025-06-10:/neurodesktop-storage/test_workflow$ module load fsl/6.0.7.1
(base) jovyan@neurodesktop-2025-06-10:/neurodesktop-storage/test_workflow$ which bet
/cvmfs/neurodesk.ardc.edu.au/containers/fsl_6.0.7.1_20230809/bet
(base) jovyan@neurodesktop-2025-06-10:/neurodesktop-storage/test_workflow$ bet -h
FATAL:   while loading overlay images: you can't specify --writable-tmpfs with another writable overlay image (/tmp/apptainer_overlay)

Could you please advise what the recommended workaround is for CentOS-7.9 deployments?

[stebo85]

interesting!

So, you run the neurodesktop image via docker?

can you run and report back:

cat /cvmfs/neurodesk.ardc.edu.au/containers/fsl_6.0.7.1_20230809/bet

echo $neurodesk_singularity_opts

then try and run:

export neurodesk_singularity_opts=""
bet

does it work then?

[stebo85]

Dear @B8017913,

To exclude firewall problems on your side, can you include this parameter in your docker command?

-e CVMFS_DISABLE=true \

this will disable the container streaming and you need to download the containers on first use through the application menu.

[B8017913]

Thank you for your kind advice. I have tried what you recommended.

When I attempted to open the mricrogl_GUI application via the GUI interface, the same error message appeared in the bash interface during startup:

[INFO] fetch_containers.sh: testing if the container runs:
WARNING: SINGULARITY_BINDPATH and APPTAINER_BINDPATH have different values, using the latter
FATAL:   while loading overlay images: you can't specify --writable-tmpfs with another writable overlay image (/tmp/apptainer_overlay)
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
the container is incomplete and needs to be re-downloaded. You could try:
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
rm -rf /neurocommand/local/containers/mricrogl_1.2.20211006_*
rm -rf /neurocommand/local/containers/modules/mricrogl/1.2.20211006
Press any key to exit...

Ather applications such as fsl:

[INFO] fetch_containers.sh: executing run_transparent_singularity.sh --container fsl_6.0.7.16_20250131.simg in /neurocommand/local/containers/fsl_6.0.7.16_20250131
[DEBUG] This is the run_transparent_singularity.sh script
-------------------------------------
installing container fsl_6.0.7.16_20250131.simg
-------------------------------------
-------------------------------------
IMPORTANT: you need to set your system specific mount points in your .bashrc!: e.g. export SINGULARITY_BINDPATH="/opt,/data"
-------------------------------------
containerName: fsl
containerVersion: 6.0.7.16
containerDate: 20250131
containerEnding: simg
checking if fsl_6.0.7.16_20250131.simg exists in the cvmfs cache ...
fsl_6.0.7.16_20250131.simg does not exists in cvmfs. Testing Nectar temporary Object storage next: 

Neuroimaging tools are accessible via the Neurodesktop Applications menu and running them through the menu will provide help and setup instructions. If you are familiar with the tools and you want to combine multiple tools in one script, you can run "ml av" to see which tools are available and then use "ml <tool>/<version>" to load them. 
CVMFS is disabled. Using local containers stored in /neurodesktop-storage/containers/modules/
(base) jovyan@neurodesktop-2025-06-10:~$ echo $PATH
/opt/conda/bin:/opt/conda/condabin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jovyan/.local/bin:/opt/conda/bin:/opt/conda/condabin:/home/jovyan/.local/bin:/opt/conda/bin:/opt/conda/condabin
(base) jovyan@neurodesktop-2025-06-10:~$ module load fsl/6.0.7.1
(base) jovyan@neurodesktop-2025-06-10:~$ echo $PATH
/neurodesktop-storage/containers/fsl_6.0.7.1_20230809:/opt/conda/bin:/opt/conda/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/jovyan/.local/bin
(base) jovyan@neurodesktop-2025-06-10:~$ bet -h
bash: bet: command not found

After realizing this might be related to network firewall issues with the lab server, I suspect that the lab network has very strict firewall restrictions for security reasons, which prevent direct access to external networks. Through testing, I can only ping github.com and cannot connect to Docker Hub.

While disabling CVMFS, I also tried reading through materials related to NeuroContainer, and attempted to pull tool images on my local PC for the NeuroDesktop image to access and use. However, I am not quite sure how to perform this operation. Could you please provide some suggestions?

[stebo85]

@Vbitz - can you arrange a call with them and figure out what's happening?

[Vbitz]

Yes of course. @B8017913 my email address is j.scarsbrook@uq.edu.au. Can you send me an email so we can organise a call to debug the issue.

[B8017913]

Thank you so much! The issue I encountered may not just be related to the network or firewall. I'll email you to describe the problem ASAP.