Do we need to set umask 0o022? Not doing so could cause subsequent issue: jupyterhub-singleuser: not found

[tinlyx]

I just installed TLJH on a Ubuntu server 18.04, falling the steps in http://tljh.jupyter.org/en/latest/install/custom-server.html. Note that this is a Ubuntu Server (not Desktop) probably without X-windows etc.

I was able to see the login interface and login to create the first admin user. But after that, the server always gives a Spawn failed: Timeout error. Detailed logs are at the bottom. It seems the error is:

Started /bin/bash -c cd /home/jupyter-admin && exec jupyterhub-singleuser --port=40967.
Aug 13 12:22:39 
mysvr bash[2492]: /bin/bash: line 0: exec: jupyterhub-singleuser: not found

I tried to edit /etc/systemd/system/jupyterhub.service and add the /opt/tljh/user/bin to the PATH, to no avail.

I did the same on a LUbuntu 18.04 Desktop VM, and everything was fine.

I was wondering if the installation instructions should be any different for Ubuntu server?
And how should this error be fixed?

Thanks,

systemd logs:

admin@mysvr:~$ sudo journalctl -b -u jupyterhub
-- Logs begin at Sat 2019-07-20 17:00:30 UTC, end at Tue 2019-08-13 12:23:34 UTC. --
Aug 13 12:21:37 mysvr systemd[1]: Started jupyterhub.service.
Aug 13 12:21:39 mysvr python3[1231]: [I 2019-08-13 12:21:39.385 JupyterHub app:2120] Using Authenticator: firstuseauthenticator.firstusea
Aug 13 12:21:39 mysvr python3[1231]: [I 2019-08-13 12:21:39.385 JupyterHub app:2120] Using Spawner: builtins.UserCreatingSpawner
Aug 13 12:21:39 mysvr python3[1231]: [I 2019-08-13 12:21:39.414 JupyterHub app:1257] Loading cookie_secret from /opt/tljh/state/jupyterhu
Aug 13 12:21:39 mysvr python3[1231]: [I 2019-08-13 12:21:39.520 JupyterHub app:1563] Not using whitelist. Any authenticated user will be 
Aug 13 12:21:39 mysvr python3[1231]: [I 2019-08-13 12:21:39.542 JupyterHub app:2337] Hub API listening on http://127.0.0.1:15001/hub/
Aug 13 12:21:39 mysvr python3[1231]: [I 2019-08-13 12:21:39.543 JupyterHub app:2352] Not starting proxy
Aug 13 12:21:39 mysvr python3[1231]: [I 2019-08-13 12:21:39.543 JupyterHub proxy:319] Checking routes
Aug 13 12:21:39 mysvr python3[1231]: [I 2019-08-13 12:21:39.543 JupyterHub app:2422] JupyterHub is now running at http://:8000
Aug 13 12:22:38 mysvr python3[1231]: [I 2019-08-13 12:22:38.777 JupyterHub log:174] 302 GET / -> /hub/ (@10.125.159.114) 2.39ms
Aug 13 12:22:38 mysvr python3[1231]: [I 2019-08-13 12:22:38.948 JupyterHub log:174] 302 GET /hub/ -> /hub/spawn (admin@10.125.159.114)
Aug 13 12:22:39 mysvr python3[1231]: Running as unit: jupyter-admin.service
Aug 13 12:22:40 mysvr python3[1231]: [I 2019-08-13 12:22:40.047 JupyterHub log:174] 302 GET /hub/spawn -> /hub/spawn-pending/admin (ju
Aug 13 12:22:40 mysvr python3[1231]: [I 2019-08-13 12:22:40.146 JupyterHub pages:303] admin is pending spawn
Aug 13 12:22:40 mysvr python3[1231]: [I 2019-08-13 12:22:40.260 JupyterHub log:174] 200 GET /hub/spawn-pending/admin (admin@10.125.
Aug 13 12:22:49 mysvr python3[1231]: [W 2019-08-13 12:22:49.046 JupyterHub base:900] User admin is slow to start (timeout=10)
admin@mysvr:~$ sudo journalctl -b -u jupyter-admin 
-- Logs begin at Sat 2019-07-20 17:00:30 UTC, end at Tue 2019-08-13 12:24:04 UTC. --
Aug 13 12:22:39 mysvr systemd[1]: Started /bin/bash -c cd /home/jupyter-admin && exec jupyterhub-singleuser --port=40967.
Aug 13 12:22:39 mysvr bash[2492]: /bin/bash: line 0: exec: jupyterhub-singleuser: not found
Aug 13 12:22:39 mysvr systemd[1]: jupyter-admin.service: Main process exited, code=exited, status=127/n/a
Aug 13 12:22:39 mysvr systemd[1]: jupyter-admin.service: Failed with result 'exit-code'.

[rtadewald]

Same issue here...

[robnagler]

Still an issue. This does not work:

vagrant init ubuntu/bionic64
vagrant up
vagrant ssh
sudo su -
apt install python3 python3-dev git curl
curl https://raw.githubusercontent.com/jupyterhub/the-littlest-jupyterhub/master/bootstrap/bootstrap.py | sudo -E python3 - --admin vagrant

[robnagler]

My problem was the umask was set incorrectly. The above instructions do work.

[aduenner]

@robnagler #554 worked for me.

Steps to resolve the issue:

1. Completely uninstall tljh using the steps listed in http://tljh.jupyter.org/en/latest/topic/installer-actions.html and reboot
2. download latest boostrap.py via:
   curl https://raw.githubusercontent.com/jupyterhub/the-littlest-jupyterhub/master/bootstrap/bootstrap.py -O
3. Edit bootstrap.py by adding os.umask(0o022) as shown in the pull request
4. sudo -E python3 boostrap.py --admin <adminuser>

[consideRatio]

This issue can perhaps be resolved by a one-liner: #554 (comment)

[consideRatio]

Background to question

When TLJH automatically installs some packages, it subsequently runs fix_permissions(prefix). This was implemented in #268 where @minrk wrote:

conda and pip may not guarantee installed files are not world-writeable

the-littlest-jupyterhub/tljh/conda.py

Lines 130 to 140 in 2d6d970

	def ensure_pip_packages(prefix, packages, upgrade=False):
	"""
	Ensure pip packages are installed in the given conda prefix.
	"""
	abspath = os.path.abspath(prefix)
	pip_executable = [os.path.join(abspath, 'bin', 'python'), '-m', 'pip']
	pip_cmd = pip_executable + ['install']
	if upgrade:
	pip_cmd.append('--upgrade')
	utils.run_subprocess(pip_cmd + packages)
	fix_permissions(prefix)

Question

Shouldn't we do something to ensure a normal user that does a sudo -E pip install ... also applies fix_permissions(prefix)? Wouldn't the users otherwise run into issues like we resolved for the times when TLJH itself installs packages?

Is that something to apply a 0o022 umask?

---

@minrk @yuvipanda this is beyond my knowledge domain. I've never considered umask before and would love to have some experienced input about this.

#554 suggested setting a umask explicitly to 0o022 for when the bootstrap.py + TLJH python code is executing, but I figure it would also need to be set so it influences the user environments where they may do sudo -E pip install ... if so. Well, I'm out of my depth reasoning about this. Any help greatly appreciated.