Another case of "Exception occurred in preexec_fn" when running as non-root-user

[MincingWords]

How to reproduce the issue
Execute './jupyterhub" in ~/.local/bin as non-root user

What you expected to happen
Not get 500 errors when regular user logs into jupyterhub

What actually happens
get 500 errors when regular user logs into jupyterhub

Share what version of JupyterHub you are using
0.8.1
Running jupyter troubleshoot from the command line, if possible, and posting
its output would also be helpful.

[NOTE: user1 ran "./jupyter troubleshoot" in ~/.local/bin]

$PATH:
.
/usr/x86_64-pc-linux-gnu/gcc-bin/5.4.0
/usr/local/bin
/usr/bin
/bin
/opt/bin
/usr/src/spark-2.2.0/bin

sys.path:
/home/juphub/.local/bin
/usr/lib64/python34.zip
/usr/lib64/python3.4
/usr/lib64/python3.4/plat-linux
/usr/lib64/python3.4/lib-dynload
/home/user1/.local/lib64/python3.4/site-packages
/usr/lib64/python3.4/site-packages

sys.executable:
/usr/bin/python3.4

sys.version:
3.4.5 (default, Nov 5 2017, 21:56:39)
[GCC 5.4.0]

platform.platform():
Linux-4.12.12-gentoo-x86_64-Intel-R-_Core-TM-i7-3520M_CPU@_2.90GHz-with-gentoo-2.4.1

which -a jupyter:
/home/juphub/.local/bin/jupyter

pip list:
alembic (0.9.6)
appdirs (1.4.3)
backports-abc (0.5)
bleach (2.1.1)
certifi (2017.4.17)
chardet (3.0.4)
decorator (4.1.2)
entrypoints (0.2.3)
gdbus-codegen (2.50.3)
gentoolkit (0.4.0)
html5lib (1.0b10)
idna (2.6)
ipykernel (4.6.1)
ipython (6.2.1)
ipython-genutils (0.2.0)
java-config (2.2.0)
jedi (0.11.0)
Jinja2 (2.9.6)
jsonschema (2.6.0)
jupyter-client (5.1.0)
jupyter-core (4.4.0)
jupyterhub (0.8.1)
Mako (1.0.7)
MarkupSafe (1.0)
mistune (0.8.1)
nbconvert (5.3.1)
nbformat (4.4.0)
notebook (5.2.1)
packaging (16.8)
pamela (0.3.0)
pandocfilters (1.4.2)
parso (0.1.0)
pexpect (4.2.1)
pickleshare (0.7.4)
pip (9.0.1)
portage (2.3.8)
prompt-toolkit (1.0.15)
ptyprocess (0.5.2)
Pygments (2.2.0)
pyparsing (2.2.0)
python-dateutil (2.6.1)
python-editor (1.0.3)
python-oauth2 (1.0.1)
pyxattr (0.5.5)
pyzmq (16.0.3)
requests (2.18.4)
setuptools (34.0.2)
simplegeneric (0.8.1)
six (1.10.0)
SQLAlchemy (1.1.15)
sudospawner (0.5.1)
terminado (0.6)
testpath (0.3.1)
tornado (4.5.2)
traitlets (4.3.2)
typing (3.6.2)
urllib3 (1.22)
wcwidth (0.1.7)
webencodings (0.5.1)

Here is the log output of jupyterhub when user1 tries to log in:

[I 2017-11-08 16:30:56.098 JupyterHub log:122] 200 GET /hub/home (user1@10.224.61.209) 11.17ms
[I 2017-11-08 16:30:57.886 JupyterHub login:19] User logged out: user1
[I 2017-11-08 16:30:57.890 JupyterHub log:122] 302 GET /hub/logout → /hub/login (user1@10.224.61.209) 4.39ms
[I 2017-11-08 16:30:57.907 JupyterHub log:122] 200 GET /hub/login (@10.224.61.209) 9.39ms
[I 2017-11-08 16:31:04.334 JupyterHub base:346] User logged in: user1
[I 2017-11-08 16:31:04.391 JupyterHub spawner:978] Spawning jupyterhub-singleuser --port=41789
[E 2017-11-08 16:31:04.399 JupyterHub user:427] Unhandled error starting user1's server: Exception occurred in preexec_fn.
[E 2017-11-08 16:31:04.419 JupyterHub web:1590] Uncaught exception POST /hub/login?next= (10.224.61.209)
    HTTPServerRequest(protocol='http', host='10.224.60.204:8000', method='POST', uri='/hub/login?next=', version='HTTP/1.1', remote_ip='10.224.61.209', headers={'X-Forwarded-Proto': 'http', 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', 'X-Forwarded-Port': '8000', 'Content-Type': 'application/x-www-form-urlencoded', 'Content-Length': '29', 'X-Forwarded-Host': '10.224.60.204:8000', 'Referer': 'http://10.224.60.204:8000/hub/login', 'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:56.0) Gecko/20100101 Firefox/56.0', 'Upgrade-Insecure-Requests': '1', 'Accept-Encoding': 'gzip, deflate', 'Accept-Language': 'en-US,en;q=0.5', 'X-Forwarded-For': '10.224.61.209', 'Host': '10.224.60.204:8000', 'Connection': 'close'})
    Traceback (most recent call last):
      File "/home/juphub/.local/lib64/python3.4/site-packages/tornado/web.py", line 1511, in _execute
        result = yield result
      File "/home/juphub/.local/lib64/python3.4/site-packages/jupyterhub/handlers/login.py", line 94, in post
        yield self.spawn_single_user(user)
      File "/home/juphub/.local/lib64/python3.4/site-packages/jupyterhub/handlers/base.py", line 475, in spawn_single_user
        yield gen.with_timeout(timedelta(seconds=self.slow_spawn_timeout), finish_spawn_future)
      File "/home/juphub/.local/lib64/python3.4/site-packages/jupyterhub/handlers/base.py", line 445, in finish_user_spawn
        yield spawn_future
      File "/home/juphub/.local/lib64/python3.4/site-packages/jupyterhub/user.py", line 439, in spawn
        raise e
      File "/home/juphub/.local/lib64/python3.4/site-packages/jupyterhub/user.py", line 378, in spawn
        ip_port = yield gen.with_timeout(timedelta(seconds=spawner.start_timeout), f)
      File "/home/juphub/.local/lib64/python3.4/site-packages/jupyterhub/spawner.py", line 988, in start
        self.proc = Popen(cmd, **popen_kwargs)
      File "/usr/lib64/python3.4/subprocess.py", line 856, in __init__
        restore_signals, start_new_session)
      File "/usr/lib64/python3.4/subprocess.py", line 1461, in _execute_child
        raise child_exception_type(err_msg)
    subprocess.SubprocessError: Exception occurred in preexec_fn.
    
[E 2017-11-08 16:31:04.423 JupyterHub log:114] {
      "X-Forwarded-Proto": "http",
      "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
      "X-Forwarded-Port": "8000",
      "Content-Type": "application/x-www-form-urlencoded",
      "Content-Length": "29",
      "X-Forwarded-Host": "10.224.60.204:8000",
      "Referer": "http://10.224.60.204:8000/hub/login",
      "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:56.0) Gecko/20100101 Firefox/56.0",
      "Upgrade-Insecure-Requests": "1",
      "Accept-Encoding": "gzip, deflate",
      "Accept-Language": "en-US,en;q=0.5",
      "X-Forwarded-For": "10.224.61.209",
      "Host": "10.224.60.204:8000",
      "Connection": "close"
    }
[E 2017-11-08 16:31:04.424 JupyterHub log:122] 500 POST /hub/login?next= (@10.224.61.209) 119.75ms

Gentoo Linux is a little odd in that it doesn't let regular users or root run pip install globally; it has to be run with --user on the end so that package files wind up in ~/.local. My user "juphub" is your "rhea" and "user1" is your "zoe" or "wash". So juphub has run pip install jupyterhub and so forth. In /home/user1 we have this:

lrwxrwxrwx 1 user1 user1 19 Nov 8 11:16 .local -> /home/juphub/.local

and /home/juphub/.local and contents are chmodded 755 except for /home/juphub/.local/bin/jupyterhub_cookie_secret which is 700.

In /etc/sudoers I have this:

# the command(s) the Hub can run on behalf of the above users without needing a password
# the exact path may differ, depending on how sudospawner was installed
Cmnd_Alias JUPYTER_CMD = /home/juphub/.local/bin/sudospawner
Cmnd_Alias JUPYTER_CMD_1 = /home/user1/.local/bin/sudospawner


# actually give the Hub user permission to run the above command on behalf
# of the above users without prompting for a password
# ADD ADD'L USERS TO THE JUPYTERHUB GROUP FOR THEM TO BE INCLUDED HERE
juphub ALL=(%jupyterhub) NOPASSWD:JUPYTER_CMD
juphub ALL=(%jupyterhub) NOPASSWD:JUPYTER_CMD_1

/etc/group contains

shadow:x:1003:juphub,user1
jupyterhub:x:1004:juphub,user1

User user1 can execute "sudo -u juphub sudo -n -u $USER /home/juphub/.local/bin/sudospawner --help" and it works.

[minrk]

JupyterHub doesn't user a per-user sudospawner path. It will always use the same sudospawner executable, which must be readable and executable by all users. Installing JupyterHub in a single user's ~/.local is likely to run into permission issues.

Try making a virtualenv outside the user's home directory and installing JupyterHub there, then verify that all users can read and run those files. It still can be in the home directory, but do verify that all users can launch Python in that env.

However, the error in your log suggests that perhaps sudospawner isn't being used, instead it looks like the default LocalProcessSpawner is being used. Can you share your jupyterhub_config.py as well?

[MincingWords]

Thanks for your response. Right now my jupyterhub_config.py is nothing but comments (I forget the jupyterhub option to generate one, but that's all I've done plus moving it to /etc/jupyterhub. I've seen postings about invoking jupyterhub with --JupyterHub.spawner_class=sudospawner.SudoSpawner and I've tried that, but when I do and try to log into jupyterhub as user1 this happens:

[I 2017-11-09 12:28:20.798 JupyterHub base:346] User logged in: user1
sudo: a password is required
[E 2017-11-09 12:28:20.901 JupyterHub spawner:80] Failed to get JSON result from mediator: ''
[E 2017-11-09 12:28:20.903 JupyterHub user:427] Unhandled error starting user1's server: substring not found
[E 2017-11-09 12:28:20.937 JupyterHub web:1590] Uncaught exception POST /hub/login?next= (10.224.61.209)
    HTTPServerRequest(protocol='http', host='10.224.60.204:8000', method='POST', uri='/hub/login?next=', version='HTTP/1.1', remote_ip='10.224.61.209', headers={'X-Forwarded-Proto': 'http', 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', 'Content-Type': 'application/x-www-form-urlencoded', 'X-Forwarded-Host': '10.224.60.204:8000', 'Accept-Encoding': 'gzip, deflate', 'Upgrade-Insecure-Requests': '1', 'Content-Length': '29', 'Connection': 'close', 'X-Forwarded-For': '10.224.61.209', 'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:56.0) Gecko/20100101 Firefox/56.0', 'Referer': 'http://10.224.60.204:8000/hub/login', 'Accept-Language': 'en-US,en;q=0.5', 'Host': '10.224.60.204:8000', 'X-Forwarded-Port': '8000'})
    Traceback (most recent call last):
      File "/home/juphub/.local/lib64/python3.4/site-packages/tornado/web.py", line 1511, in _execute
        result = yield result
      File "/home/juphub/.local/lib64/python3.4/site-packages/jupyterhub/handlers/login.py", line 94, in post
        yield self.spawn_single_user(user)
      File "/home/juphub/.local/lib64/python3.4/site-packages/jupyterhub/handlers/base.py", line 475, in spawn_single_user
        yield gen.with_timeout(timedelta(seconds=self.slow_spawn_timeout), finish_spawn_future)
      File "/home/juphub/.local/lib64/python3.4/site-packages/jupyterhub/handlers/base.py", line 445, in finish_user_spawn
        yield spawn_future
      File "/home/juphub/.local/lib64/python3.4/site-packages/jupyterhub/user.py", line 439, in spawn
        raise e
      File "/home/juphub/.local/lib64/python3.4/site-packages/jupyterhub/user.py", line 378, in spawn
        ip_port = yield gen.with_timeout(timedelta(seconds=spawner.start_timeout), f)
      File "/home/juphub/.local/lib64/python3.4/site-packages/sudospawner/spawner.py", line 89, in start
        reply = yield self.do(action='spawn', args=self.get_args(), env=self.get_env())
      File "/home/juphub/.local/lib64/python3.4/site-packages/sudospawner/spawner.py", line 77, in do
        data_str = data_str[data_str.index('{'):data_str.rindex('}')+1]
    ValueError: substring not found

Yet user1 can shell in and go "sudo -u juphub sudo -n -u $USER /home/juphub/.local/bin/sudospawner --help" and ti works.

[MincingWords]

UPDATE: It's a long story but I was just now able to get user1's login to work.

[rcasey-earthscope]

I was disappointed to find that this was closed without elaboration after the user solved their problem. Others of us that have the exact same problem are looking for an answer or direction of some sort. Having the shell sudospawner call working through the sudo redirects is a good sign that we are just a couple of configuration steps away from getting this to work. It's likely that it's not enough to just set the spawner_class, but also set the spawner cmd call in jupyterhub_config.py:

c.JupyterHub.spawner_class = 'sudospawner.SudoSpawner'
c.Spawner.cmd = '/home/juphub/.local/bin/sudospawner'

This conduit will allow a non-privileged user to perform a single-user spawn via sudospawner.

[mdivk]

I echo rcasey-iris, I am having the same issue here, I hope there would be an official and more elegant solution on this issue, but I am not sure if this "closed" issue would even get attention, anyway, I hope so.

[VidJa]

same issue here on a fresh U18.04 install. I have my old 16.04 working configuration at hand, but I get the exact same error as @MincingWords

never mind: my sudospawner was not executable by the jupyterhub user