Update jupyterhub_config.py.j2 for JupyterHub 3.1.1 & /run/jupyterhub-proxy.pid [+ refine doc. Is error "IOPub data rate exceeded" common &/or solvable?]

[holta]

Auto-customizing /opt/iiab/jupyterhub/etc/jupyterhub/jupyterhub_config.py in a readable/meaningful way would be too complex, and not worth it IMO given frequent upstream changes — to this 1500+ line configuration file and its in-line guidance/explanations.

So this PR is an update to IIAB's templates/jupyterhub_config.py.j2 almost 2 years later — for JupyterHub 3.1.1+ and also to specify a cleaner location for /run/jupyterhub-proxy.pid as discussed at:

• BUG: Find a better place for pid files #3475

Tangentially related:

• PR Block iiab-install of JupyterHub & Calibre-Web on 32-bit OS's, til RasPiOS fixes Rust, wheels / cryptography #3422

[holta]

Tested successfully with http://box/jupyterhub (which shows JupyterLab 3.6.1) on a Ubuntu Server 22.04.1 VM:

root@box:~# ls -l /run/jupyterhub-proxy.pid
-rw-r--r-- 1 root root 5 Feb  8 11:50 /run/jupyterhub-proxy.pid

[holta]

Summary of this PR...from the top of the new roles/jupyterhub/templates/jupyterhub_config.py.j2 :

# [2023-02-10] /opt/iiab/jupyterhub/etc/jupyterhub/jupyterhub_config.py update:
# https://jupyterhub.readthedocs.io/en/stable/getting-started/config-basics.html
#
# 1) To generate this 1500+ line stub, I first ran JupyterHub 3.1.1's:
#    /opt/iiab/jupyterhub/bin/jupyterhub --generate-config
# 2) Then I manually inserted [8, formerly 9] of IIAB's 10 legacy custom lines below, from:
#    grep ^c /opt/iiab/iiab/roles/jupyterhub/templates/jupyterhub_config.py.j2
# 3) Finally I added the following line on @jvonau's suggestion: (#3475)
#    c.ConfigurableHTTPProxy.pid_file = "/run/jupyterhub-proxy.pid"

[holta]

@cwivagg can you give this PR a quick review if you have a moment?

[holta]

Tested successfully with http://box/jupyterhub (which shows JupyterLab 3.6.1) on a Ubuntu Server 22.04.1 VM:

Also spot-tested on the latest daily build of Ubuntu Server 23.04 ✔️

[holta]

OLD file for the record:

# grep -n ^c /opt/iiab/iiab/roles/jupyterhub/templates/jupyterhub_config.py.j2
67:c.JupyterHub.admin_access = True
117:c.JupyterHub.authenticator_class = 'firstuseauthenticator.FirstUseAuthenticator'
127:c.JupyterHub.base_url = '/jupyterhub'
197:c.JupyterHub.cookie_secret = b'helloiiabitsrainingb123456789012'
214:c.JupyterHub.db_url = 'sqlite:///{{ jupyterhub_venv }}/jupyterhub.sqlite'
592:c.JupyterHub.spawner_class = 'systemdspawner.SystemdSpawner'
1060:c.Authenticator.admin_users = set(['admin'])
1061:c.Authenticator.dbm_path = "{{ jupyterhub_venv }}/etc/passwords.dbm"
1237:c.SystemdSpawner.dynamic_users = True
1238:c.SystemdSpawner.user_workingdir = '/opt/iiab/notebooks/{USERNAME}'

NEW file for the record: [REVISED 2023-08-09]

# grep -n ^c /opt/iiab/iiab/roles/jupyterhub/templates/jupyterhub_config.py.j2
14:c = get_config()  #noqa
190:c.JupyterHub.authenticator_class = 'firstuseauthenticator.FirstUseAuthenticator'
201:c.JupyterHub.base_url = '/jupyterhub'
274:c.JupyterHub.cookie_secret = b'helloiiabitsrainingb123456789012'
310:c.JupyterHub.db_url = 'sqlite:///{{ jupyterhub_venv }}/jupyterhub.sqlite'
667:c.ConfigurableHTTPProxy.pid_file = "/run/jupyterhub-proxy.pid"
791:c.JupyterHub.spawner_class = 'systemdspawner.SystemdSpawner'
792:c.SystemdSpawner.dynamic_users = True
1333:c.Authenticator.admin_users = set(['admin'])
1334:c.Authenticator.dbm_path = "{{ jupyterhub_venv }}/etc/passwords.dbm"

[holta]

@deldesir asks if the equivalent of --NotebookApp.iopub_data_rate_limit=1.0e10 (would this be c.NotebookApp.iopub_data_rate_limit = 1.0e10 ?) should be added to jupyterhub_config.py.j2 so that he no longer gets this error:

IOPub data rate exceeded.
The notebook server will temporarily stop sending output
to the client in order to avoid crashing it.

As recommended by: https://stackoverflow.com/questions/43288550/iopub-data-rate-exceeded-in-jupyter-notebook-when-viewing-image

(He'll look into it overnight.)

[deldesir]

This is correct for me. The "IOPub data rate exceeded" error often occurs when a function or command produces repetitive output more than a thousand times. I usually get around it by passing --ServerApp.iopub_data_rate_limit 10000000000 at the command line (the higher number does the trick).

[deldesir]

@deldesir asks if the equivalent of --NotebookApp.iopub_data_rate_limit=1.0e10 (would this be c.NotebookApp.iopub_data_rate_limit = 1.0e10 ?) should be added to jupyterhub_config.py.j2 so that he no longer gets this error:

IOPub data rate exceeded.
The notebook server will temporarily stop sending output
to the client in order to avoid crashing it.

As recommended by: https://stackoverflow.com/questions/43288550/iopub-data-rate-exceeded-in-jupyter-notebook-when-viewing-image

(He'll look into it overnight.)

Yes. In my case (and maybe others) it would be good to set it in the configuration once and for all.

[holta]

In my case (and maybe others) it would be good to set it in the configuration once and for all.

@deldesir is the exact string c.NotebookApp.iopub_data_rate_limit = 1.0e10 within /opt/iiab/jupyterhub/etc/jupyterhub/jupyterhub_config.py tested to solve your "IOPub data rate exceeded" problem?

[holta]

@deldesir the following JupyterHub config file setting...

c.Spawner.args = ['--NotebookApp.iopub_data_rate_limit=1000000000']

...is suggested by:

• IOPub limits setting to jupyterhub jupyterhub/jupyterhub#1152 (comment)

(Can you test & confirm this is truly what you need?)

[holta]

@georgejhunt it looks like you set c.SystemdSpawner.user_workingdir = '/opt/iiab/notebooks/{USERNAME}' that's currently on Line 793 of jupyterhub_config.py.p2

Do you know if this is meant to match Individual student folders are created in /var/lib/private/ as confirmed in an actual IIAB JupyterHub test, and mentioned in jupyterhub/README.md ?

(Or is /opt/iiab/notebooks/{USERNAME} somehow also used and necessary?!)

[deldesir]

@deldesir the following JupyterHub config file setting...

c.Spawner.args = ['--NotebookApp.iopub_data_rate_limit=1000000000']

(Can you test & confirm this is truly what you need?)

I tested it. It has no effect.

[holta]

@georgejhunt it looks like you set c.SystemdSpawner.user_workingdir = '/opt/iiab/notebooks/{USERNAME}' that's currently on Line 793 of jupyterhub_config.py.p2

1. Above is documented at https://github.com/jupyterhub/systemdspawner#user_workingdir if we still need the following?

   c.SystemdSpawner.user_workingdir = '/opt/iiab/notebooks/{USERNAME}'
   

2. Meanwhile several tickets recommend a separate (additional???) setting like...

   c.Spawner.notebook_dir = '/data/jupyterhub/jupyterhub_notebooks'
   

   ...currently commented out on Line 1160 of jupyterhub_config.py.j2

QUESTION: Are either-or-both above dir settings advisable?

[holta]

1. Above is documented at https://github.com/jupyterhub/systemdspawner#user_workingdir if we still need the following?

   c.SystemdSpawner.user_workingdir = '/opt/iiab/notebooks/{USERNAME}'
   

As explained in the systemdspawner#user_workingdir doc, the above IIAB setting is completely ignored because of c.SystemdSpawner.dynamic_users = True on the line just above it (Line 792 of jupyterhub_config.py.j2).

PS c.SystemdSpawner.dynamic_users = True is explained in these 2 places:

• https://github.com/jupyterhub/systemdspawner#local-users
• https://github.com/jupyterhub/systemdspawner#dynamic_users

[holta]

c.Spawner.args = ['--NotebookApp.iopub_data_rate_limit=1000000000']

(Can you test & confirm this is truly what you need?)

I tested it. It has no effect.

You should probably mention that at jupyterhub/jupyterhub#1152 as ~18 people there (suggest) it is/was working for them.

(And if possible, be specific as to how you tested it.)

[holta]

@deldesir please post a separate PR with a fix to your iopub_data_rate_limit issue, so this PR can now move forward without delay, Thanks!

[deldesir]

No problem.