Mounting Large Volume #82
Replies: 0 comments · 32 replies
|
Dear @etmaddox-creator - can you share more details on how you are mounting that drive exactly? Did you try it via the mount options of docker? Have you tried mounting it after the hub started? Is this nfs drive mapped on your Linux machine and you are trying to pass it through to docker? |
All reactions
|
Yes, the drive is mapped on the Linux machine and I am trying to make it accessible to the Neurodesktop container. To do so, I've added it as an additional data directory in the Neurodesktop settings prior to actually creating the Desktop environment. I've given Docker access to the drive and once the container is created it shows up under the "bind mount" section of the container settings within Docker Desktop. Once I try to start the desktop environment, it hangs on the "creating new session" screen before timing out after 40 minutes (I think that's what the logs show, I've attached them below). I'm not 100% what you mean by trying to mount it after the hub started. Does this answer that question? If not, how can I do that? Thanks for the response! [2025-12-12 09:55:59.441] [info] In production mode \n' +
' \n' + ' \n' + '\n' + ' \n' + ' \n' + ' <style>\n' + ' .st0 {\n' + ' fill: none;\n' + ' stroke-width: 5px;\n' + ' }\n' + '\n' + ' .st0, .st1 {\n' + ' stroke: #fff;\n' + ' stroke-miterlimit: 10;\n' + ' }\n' + '\n' + ' .st2 {\n' + ' fill: #6aa329;\n' + ' }\n' + '\n' + ' .st3, .st1 {\n' + ' fill: #fff;\n' + ' }\n' + '\n' + ' .st4 {\n' + ' fill: #101806;\n' + ' }\n' + ' </style>\n' + ' \n' + ' \n' + ' <animateMotion\n' + ' path="M0,0 0 90 0 0"\n' + ' begin="0s" dur="4s" repeatCount="indefinite"\n' + ' />\n' + ' \n' + ' <animateMotion\n' + ' path="M0,0 0 90 0 0"\n' + ' begin="0s" dur="4s" repeatCount="indefinite"\n' + ' />\n' + ' <animateMotion\n' + ' path="M0,0 0 90 0 0"\n' + ' begin="0s" dur="4s" repeatCount="indefinite"\n' + ' />\n' + ' <animateMotion\n' + ' path="M0,0 0 90 0 0"\n' + ' begin="0s" dur="4s" repeatCount="indefinite"\n' + ' />\n' + ' <animateMotion\n' + ' path="M0,0 0 90 0 0"\n' + ' begin="0s" dur="4s" repeatCount="indefinite"\n' + ' />\n' + ' <animateMotion\n' + ' path="M0,0 0 90 0 0"\n' + ' begin="0s" dur="4s" repeatCount="indefinite"\n' + ' />\n' + ' <animateMotion\n' + ' path="M0,0 0 90 0 0"\n' + ' begin="0s" dur="4s" repeatCount="indefinite"\n' + ' />\n' + ' <animateMotion\n' + ' path="M0,0 0 90 0 0"\n' + ' begin="0s" dur="4s" repeatCount="indefinite"\n' + ' />\n' + ' <animateMotion\n' + ' path="M0,0 0 90 0 0"\n' + ' begin="0s" dur="4s" repeatCount="indefinite"\n' + ' />\n' + ' <animateMotion\n' + ' path="M0,0 0 90 0 0"\n' + ' begin="0s" dur="4s" repeatCount="indefinite"\n' + ' />\n' + ' <animateMotion\n' + ' path="M0,0 0 90 0 0"\n' + ' begin="0s" dur="4s" repeatCount="indefinite"\n' + ' />\n' + ' \n' + ' <animateMotion\n' + ' path="M0,0 0 90 0 0"\n' + ' begin="0s" dur="4s" repeatCount="indefinite"\n' + ' /><'... 4912 more characters, _viewReady: {} } }, closing: null, used: true } [2025-12-12 09:56:02.029] [info] token jlab:srvr:0f9776631a3b6f24c3c3cbe7bdd2f689ee8dd1 [2025-12-12 09:56:02.031] [debug] Loading config from /opt/NeurodeskApp/resources/app.asar/build/out/neurodesktop.toml... [2025-12-12 09:56:02.055] [info] isDev false [2025-12-12 09:56:02.136] [debug] !!!..... 40415 engineType docker [2025-12-12 09:56:02.141] [debug] Server launch script: [2025-12-12 09:56:04.644] [debug] setChildProcessLog a94376bbd404ef797268e2eeddb1c1b2333258f1c4bee61384345558e3e1d081 [2025-12-12 10:36:02.153] [debug] Server didn't start in time [2025-12-12 10:36:02.161] [error] (node:1385470) UnhandledPromiseRejectionWarning: TypeError: Cannot read properties of undefined (reading 'protocol') |
All reactions
|
Dear @etmaddox-creator, Let's try to find out where the problem comes from. It sounds like something is trying to change ownership of files on this mount which times out. Can you try to start the container manually without the neurodesk-app? if that works, add your mount: |
All reactions
|
That works, thank you so much!! I can access the web UI with the NFS drive mounted and it appears to be responsive. I had to change apparmor=unconfined because I kept getting an error when trying to run those commands with apparmor=neurodesktop. Could this be causing the issues? I'm running Ubuntu 22.04.5 LTS. I've tried to run the container without any apparmor changes and by adding the profile shown here. |
All reactions
|
Dear @etmaddox-creator - that's very helpful! The problem is that the Neurodesk-app appears to be performing chown operations on the additional data mount (@iishiishii - can you verify?). Ubuntu 22.04.5 shouldn't need the apparmor setting at all, so that's why it failed, but apparmor=uncfonfined will not cause any problems. We will update the neurodeskapp once we found out what the cause is and then it would be wonderful if you could test again. For now, I hope the workaround is ok. |
All reactions
|
That workaround is perfect for now, thank you so much for your help. I'm happy to test it again in the future. |
All reactions
|
@iishiishii - are you looking into this for the next release? |
All reactions
|
@jsneal1 @ethanmaddox @etmaddox-creator apologies for the delayed fix, this latest version will allow you to mount the storage without us touching on its permissions. The files inside /data have the same ownership as the host filesystem, so non-root process inside the container will be subject to host dir's permission. You can test the pre-release here |
All reactions
|
@stebo85 @iishiishii Thank you for this update. Unfortunately, the pre-release version is still giving me an error when starting up the container. [2026-04-24 15:44:21.704] [info] Logging to file (/home/etmaddox/.config/neurodeskapp/logs/main.log) at 'verbose' level [2026-04-24 16:24:24.415] [error] (node:2600228) UnhandledPromiseRejectionWarning: TypeError: Cannot read properties of undefined (reading 'protocol') |
All reactions
|
thank you for testing this - we try to replicate this and get back to you with a new version to try |
All reactions
|
and make sure to uninstall previous version and remove cache https://github.com/neurodesk/neurodesk-app/blob/master/user-guide.md#uninstalling-neurodesk-app, the latest log above doesn't seem to print out the launch script before failing so it might have old state that won't work in the latest version. |
All reactions
|
Does it matter that we are using nfs version 3 to mount the shared drive? It’s handled by /etc/fstab to mount at startup. If there’s any info we can gather for you let us know we are happy to help.
Best Regards,
Jeffrey Neal
***@***.***
LSA EHTS Desktop Support Specialist Senior
University of Michigan
… On Apr 24, 2026, at 4:41 PM, Steffen Bollmann ***@***.***> wrote:
a
|
All reactions
|
yes, most likely it is related to this because we cannot replicate the exact problem you get. we now added a few mitigations against known nfs issues and the new version is currently building. I think we will just have to test a few iterations until we get to the core of this. |
All reactions
|
@ethanmaddox @etmaddox-creator @jsneal1 : https://github.com/neurodesk/neurodesk-app/actions/runs/24911409436 - can you test this and report back with error logs? |
All reactions
|
@stebo85 It looks like this patch made some progress but the end result was still the same (timed out at 40 minutes on loading screen). Thank you for the reminder about fully uninstalling the previous version, I believe that I was doing that before but have double checked it this time. [2026-04-27 13:27:27.116] [info] In production mode |
All reactions
|
Are there any specific cached files / locations we need to delete? I would assume it writes cached data to a users profile?
Best Regards,
Jeffrey Neal
***@***.***
LSA EHTS Desktop Support Specialist Senior
University of Michigan
… On Apr 27, 2026, at 2:20 PM, etmaddox-creator ***@***.***> wrote:
@stebo85 <https://github.com/stebo85> It looks like this patch made some progress but the end result was still the same (timed out at 40 minutes on loading screen). Thank you for the reminder about fully uninstalling the previous version, I believe that I was doing that before but have double checked it this time.
[2026-04-27 13:27:27.116] [info] In production mode
[2026-04-27 13:27:27.118] [info] Logging to file (/home/etmaddox/.config/neurodeskapp/logs/main.log) at 'verbose' level
[2026-04-27 13:27:27.231] [info] startupMode welcome-page
[2026-04-27 13:27:27.232] [info] sessionConfig undefined
[2026-04-27 13:27:46.431] [info] In production mode
[2026-04-27 13:27:46.433] [info] Logging to file (/home/etmaddox/.config/neurodeskapp/logs/main.log) at 'verbose' level
[2026-04-27 13:27:46.536] [info] startupMode welcome-page
[2026-04-27 13:27:46.537] [info] sessionConfig undefined
[2026-04-27 13:27:51.243] [info] ~ createServer {
name: 'python',
path: 'C:\',
type: 'path',
versions: {},
defaultKernel: 'python3'
}
[2026-04-27 13:27:51.247] [info] token jlab:srvr:70e1f1325ef139cb7da7aa55b9fac59c125415
[2026-04-27 13:27:51.265] [info] isDev false
[2026-04-27 13:27:51.366] [info] Skipping chmod for network/remote directory: /nfs/tpolk/mind
[2026-04-27 13:27:51.366] [info] Detected nfs working directory; mounting /nfs/tpolk/mind at its native path
[2026-04-27 14:07:51.375] [error] ~ Failed to start server Error: Failed to launch Neurodesk from Promise 46447d6f9f671ecdd89c0061779a456c4aaacabb4d8b29f0f4edc79b5f42036740c57
at /opt/NeurodeskApp/resources/app.asar/build/out/main/server.js:398:28
—
Reply to this email directly, view it on GitHub <https://github.com/orgs/neurodesk/discussions/661#discussioncomment-16732863>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALYJWG7SMNHE4JY66GTYQOL4X6QGJAVCNFSM6AAAAACPSEXOLSVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTMNZTGI4DMMY>.
Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
You are receiving this because you were mentioned.
|
All reactions
|
appreciate sending the log @etmaddox-creator, could you also please send me the |
All reactions
|
Surprisingly, I was able to open the desktop environment with the nfs drive mounted! I've attached the docker and neurodesk logs below. It was very glitchy. I tried to close the container and reopen it but have gotten a new set of errors. I'm not sure why it is saying second instance, the container was deleted. There are no docker logs for this as it doesn't appear to be creating a container. New errors when trying to reopen it: [2026-04-29 13:27:00.842] [warn] [SECOND-INSTANCE] No files found in cliArgs Neurodesktop: The selectors will be substituted transparently this time, but need to be updated at source before next major release. Docker: Entered start.sh with args: start.sh jupyter lab --ServerApp.password= --no-browser --expose-app-in-browser --ServerApp.token=jlab:srvr:5b7c8c97811e71a06c74f839b945398d496637 --ServerApp.port=8888 --LabApp.quit_button=False --NotebookIntelligence.github_access_token=remember [I 2026-04-29 14:59:49.780 ServerApp] Starting up handlers.... [I 2026-04-29 14:59:49.780 ServerApp] Handler: ExampleHandler URI: /jupyterlab_slurm/get_example dict: {'log': <Logger ServerApp (DEBUG)>} [I 2026-04-29 14:59:49.780 ServerApp] Handler: UserFetchHandler URI: /jupyterlab_slurm/user dict: {'log': <Logger ServerApp (DEBUG)>} [I 2026-04-29 14:59:49.780 ServerApp] Handler: SqueueHandler URI: /jupyterlab_slurm/squeue dict: {'squeue': 'squeue', 'log': <Logger ServerApp (DEBUG)>} [I 2026-04-29 14:59:49.780 ServerApp] Handler: ScancelHandler URI: /jupyterlab_slurm/scancel dict: {'scancel': 'scancel', 'log': <Logger ServerApp (DEBUG)>} [I 2026-04-29 14:59:49.780 ServerApp] Handler: ScontrolHandler URI: /jupyterlab_slurm/scontrol/(?P.*) dict: {'scontrol': 'scontrol', 'log': <Logger ServerApp (DEBUG)>} [I 2026-04-29 14:59:49.780 ServerApp] Handler: SbatchHandler URI: /jupyterlab_slurm/sbatch dict: {'sbatch': 'sbatch', 'temporary_directory': None, 'log': <Logger ServerApp (DEBUG)>} [I 2026-04-29 14:59:49.781 ServerApp] jupyterlab_slurm | extension was successfully loaded. | | | |_ __ | | | | ___ Read the migration plan to Notebook 7 to learn about the new features and the actions to take if you are using extensions. https://jupyter-notebook.readthedocs.io/en/latest/migrate_to_notebook7.html Please note that updating to Notebook 7 might break some of your extensions. [I 2026-04-29 14:59:49.786 ServerApp] nbclassic | extension was successfully loaded.
[DEBUG] VNC setup - checking prerequisites... xrdp-sesman[24270]: [INFO ] Terminal Server Users group is disabled, allowing authentication xrdp-sesman[24270]: [INFO ] ++ created session (access granted): username jovyan, ip ::ffff:127.0.0.1:52026 - socket: 11 xrdp-sesman[24270]: [INFO ] starting Xorg session... xrdp-sesman[24270]: [INFO ] Starting session: session_pid 24668, display :10.0, width 800, height 575, bpp 24, client ip ::ffff:127.0.0.1:52026 - socket: 11, user name jovyan xrdp-sesman[24668]: [INFO ] [session start] (display 10): calling auth_start_session from pid 24668 xrdp-sesman[24270]: [ERROR] sesman_data_in: scp_process_msg failed xrdp-sesman[24668]: pam_unix(xrdp-sesman:session): session opened for user jovyan(uid=114461999) by (uid=0) xrdp-sesman[24668]: pam_systemd(xrdp-sesman:session): Failed to connect to system bus: No such file or directory xrdp-sesman[24668]: [INFO ] Found X server running at /tmp/.X11-unix/X10 xrdp-sesman[24668]: [INFO ] Session started successfully for user jovyan on display 10 xrdp-sesman[24675]: [INFO ] Starting the xrdp channel server for display 10 xrdp-sesman[24668]: [INFO ] Session in progress on display 10, waiting until the window manager (pid 24671) exits to end the session xrdp-sesman[24270]: [INFO ] Socket 11: AF_INET6 connection received from ::1 port 40994 xrdp-sesman[24270]: [INFO ] ++ reconnected session: username jovyan, display :10.0, session_pid 24668, ip ::ffff:127.0.0.1:39016 - socket: 11 xrdp-sesman[24270]: [ERROR] sesman_data_in: scp_process_msg failed xrdp-sesman[24270]: [ERROR] sesman_main_loop: trans_check_wait_objs failed, removing trans xrdp-sesman[24270]: [INFO ] Process 24845 has exited xrdp-sesman[24270]: [INFO ] Socket 11: AF_INET6 connection received from ::1 port 33004 xrdp-sesman[24270]: [INFO ] ++ reconnected session: username jovyan, display :10.0, session_pid 24668, ip ::ffff:127.0.0.1:55146 - socket: 11 xrdp-sesman[24270]: [ERROR] sesman_data_in: scp_process_msg failed xrdp-sesman[24270]: [ERROR] sesman_main_loop: trans_check_wait_objs failed, removing trans xrdp-sesman[24270]: [INFO ] Process 25075 has exited xrdp-sesman[24270]: [INFO ] Socket 11: AF_INET6 connection received from ::1 port 49316 xrdp-sesman[24270]: [INFO ] ++ reconnected session: username jovyan, display :10.0, session_pid 24668, ip ::ffff:127.0.0.1:35880 - socket: 11 xrdp-sesman[24270]: [ERROR] sesman_data_in: scp_process_msg failed xrdp-sesman[24270]: [ERROR] sesman_main_loop: trans_check_wait_objs failed, removing trans xrdp-sesman[24270]: [INFO ] Process 25157 has exited xrdp-sesman[24270]: [INFO ] Socket 11: AF_INET6 connection received from ::1 port 48484 xrdp-sesman[24270]: [INFO ] ++ reconnected session: username jovyan, display :10.0, session_pid 24668, ip ::ffff:127.0.0.1:51382 - socket: 11 xrdp-sesman[24270]: [ERROR] sesman_data_in: scp_process_msg failed xrdp-sesman[24270]: [ERROR] sesman_main_loop: trans_check_wait_objs failed, removing trans xrdp-sesman[24270]: [INFO ] Process 25258 has exited |
All reactions
|
Thanks @etmaddox-creator for sharing the log. This inconsistent behaviour is not ideal, but really appreciate your testing. We added more logging in this version https://github.com/neurodesk/neurodesk-app/releases/download/v1.15.0/NeurodeskApp-Setup-Debian-x64.deb. We want to see if the NFS drive is ready or stale (I assume it is as it's mounted at startup as you said). |
All reactions
|
Yea, I'm not sure what's causing that behavior. I tried running this version twice with the nfs drive (after confirming that it worked normally without the drive bound). It gave an error both times, but for the first attempt docker didn't show any logs because I think the container was created but not started. The second time, the container started and I copied the logs from it. Please let me know if I can provide any additional information and thanks again for all of your help with this! FIRST NEURODESK LOGS: ^[[2026-05-04 10:13:03.357] [error] Server didn't start in time SECOND NEURODESK LOGS: [2026-05-04 11:16:46.033] [error] Server didn't start in time DOCKER LOGS: |
All reactions
|
Thanks for sharing @etmaddox-creator Looking more closely at the most recent logs, the issue is tied to the container's startup latency. Did you notice a significant speed improvement when using the Docker CLI workaround that Steffen suggested previously? While we look into ways to prevent these failures, is it possible to update your |
All reactions
|
We can certainly try that. While I don’t want to post the full path from fstab I can post mount options if that helps. I can update their timeout to start and see if that helps at all.
Best Regards,
Jeffrey Neal
***@***.***
LSA EHTS Desktop Support Specialist Senior
University of Michigan
… On May 6, 2026, at 9:07 AM, Thuy Dao ***@***.***> wrote:
Thanks for sharing @etmaddox-creator <https://github.com/etmaddox-creator>
Looking more closely at the most recent logs, the issue is tied to the container's startup latency.
The system is getting stuck for nearly 40 minutes while Docker attempts to create bind mounts, which eventually triggers a timeout in the app.
Did you notice a significant speed improvement when using the Docker CLI workaround that Steffen suggested previously?
While we look into ways to prevent these failures, is it possible to update your /etc/fstab settings on the host? Reducing the timeout to timeo=150 and increasing the rsize could help the kernel recover faster and handle data more efficiently.
—
Reply to this email directly, view it on GitHub <https://github.com/orgs/neurodesk/discussions/661#discussioncomment-16827494>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALYJWG5LNPFD67RJADX27RD4ZM2HXAVCNFSM6AAAAACPSEXOLSVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTMOBSG42DSNA>.
Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
You are receiving this because you were mentioned.
|
All reactions
|
I increased the timeo to 150, here is the default rsize=131072
Best Regards,
Jeffrey Neal
***@***.***
LSA EHTS Desktop Support Specialist Senior
University of Michigan
… On May 6, 2026, at 9:07 AM, Thuy Dao ***@***.***> wrote:
Thanks for sharing @etmaddox-creator <https://github.com/etmaddox-creator>
Looking more closely at the most recent logs, the issue is tied to the container's startup latency.
The system is getting stuck for nearly 40 minutes while Docker attempts to create bind mounts, which eventually triggers a timeout in the app.
Did you notice a significant speed improvement when using the Docker CLI workaround that Steffen suggested previously?
While we look into ways to prevent these failures, is it possible to update your /etc/fstab settings on the host? Reducing the timeout to timeo=150 and increasing the rsize could help the kernel recover faster and handle data more efficiently.
—
Reply to this email directly, view it on GitHub <https://github.com/orgs/neurodesk/discussions/661#discussioncomment-16827494>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALYJWG5LNPFD67RJADX27RD4ZM2HXAVCNFSM6AAAAACPSEXOLSVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTMOBSG42DSNA>.
Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
You are receiving this because you were mentioned.
|
All reactions
|
@iishiishii I think the CLI was faster than the GUI. Thank you Jeffery for updating the fstab settings. I tried running it again after that but didn't notice any changes. |
All reactions
|
Appreciate that @jsneal1 and @etmaddox-creator |
All reactions
|
I found that with the mount, some processes freeze without retrying, so we now strictly set a timeout on docker run command to error out rather than hanging indefinitely. I also noticed it didn't wait enough for the endpoint so we extended the wait time for that. Here's the latest build. |
All reactions
|
I'm glad that you were able to replicate the issue on your end. With this version, I was able to open the jupiter hub environment but not the guacamole desktop environment without the nfs drive mounted. With it mounted, it again created the container but didn't start it. Logs are below. I also restarted docker per the logs suggestion. Also, the container is now being deleted after neurodesk fails to start or when I close the neurodesktop app. Unsure if this behavior is intended, I think it started in one of the recent builds. [2026-05-07 10:43:36.623] [info] In production mode [2026-05-07 11:23:59.442] [error] Server didn't start in time |
Uh oh!
There was an error while loading. Please reload this page.
Hello! Has anyone found success mounting a large drive (~50TB) inside Neurodesktop? I have been trying to mount an nfs drive around that size but Jupyterlab consistently times out before successful startup. I can start an instance with a smaller folder on the nfs drive, but that doesn't contain all of the data that I'm hoping to access. When I try to mount the whole drive, the container is created but does not start, and while it's possible to start this container from the command line, it's quite slow and doesn't provide access to the GUI. I am running Neurodesk with Docker on Linux. Any suggestions you all may have for this issue would be greatly appreciated!
All reactions