500 inside notebook of hailctl dataproc start cluster

[danking]

What happened?

I'm hitting a 500 error when I try to open a notebook on a dataproc cluster started with hail 0.2.116. hailctl dataproc connect mw nb will open the google bucket and I am able to see all ipynb files but when I try opening one, I'm met with the 500. I have no issue with hail 0.2.113.

Error from the logs:

May 16 14:17:07 mw116-m python[8309]: [D 14:17:07.035 NotebookApp] Path notebook/css/override.css served from /opt/conda/miniconda3/lib/python3.10/site-packages/notebook/static/notebook/css/override.css
May 16 14:17:07 mw116-m python[8309]: [E 14:17:07.041 NotebookApp] Uncaught exception GET /notebooks/gnomad-mwilson/v4/sample_qc_defs.ipynb (127.0.0.1)
May 16 14:17:07 mw116-m python[8309]:     HTTPServerRequest(protocol='http', host='localhost:8123', method='GET', uri='/notebooks/gnomad-mwilson/v4/sample_qc_defs.ipynb', version='HTTP/1.1', remote_ip='127.0.0.1')
May 16 14:17:07 mw116-m python[8309]:     Traceback (most recent call last):
May 16 14:17:07 mw116-m python[8309]:       File "/opt/conda/miniconda3/lib/python3.10/site-packages/tornado/web.py", line 1786, in _execute
May 16 14:17:07 mw116-m python[8309]:         result = await result
May 16 14:17:07 mw116-m python[8309]:       File "/opt/conda/miniconda3/lib/python3.10/site-packages/tornado/gen.py", line 786, in run
May 16 14:17:07 mw116-m python[8309]:         yielded = self.gen.send(value)
May 16 14:17:07 mw116-m python[8309]:       File "/opt/conda/miniconda3/lib/python3.10/site-packages/notebook/notebook/handlers.py", line 95, in get
May 16 14:17:07 mw116-m python[8309]:         self.write(self.render_template('notebook.html',
May 16 14:17:07 mw116-m python[8309]:       File "/opt/conda/miniconda3/lib/python3.10/site-packages/notebook/base/handlers.py", line 507, in render_template
May 16 14:17:07 mw116-m python[8309]:         return template.render(**ns)
May 16 14:17:07 mw116-m python[8309]:       File "/opt/conda/miniconda3/lib/python3.10/site-packages/jinja2/environment.py", line 1301, in render
May 16 14:17:07 mw116-m python[8309]:         self.environment.handle_exception()
May 16 14:17:07 mw116-m python[8309]:       File "/opt/conda/miniconda3/lib/python3.10/site-packages/jinja2/environment.py", line 936, in handle_exception
May 16 14:17:07 mw116-m python[8309]:         raise rewrite_traceback_stack(source=source)
May 16 14:17:07 mw116-m python[8309]:       File "/opt/conda/miniconda3/lib/python3.10/site-packages/notebook/templates/notebook.html", line 1, in top-level template code
May 16 14:17:07 mw116-m python[8309]:         {% extends "page.html" %}
May 16 14:17:07 mw116-m python[8309]:       File "/opt/conda/miniconda3/lib/python3.10/site-packages/notebook/templates/page.html", line 154, in top-level template code
May 16 14:17:07 mw116-m python[8309]:         {% block header %}
May 16 14:17:07 mw116-m python[8309]:       File "/opt/conda/miniconda3/lib/python3.10/site-packages/notebook/templates/notebook.html", line 114, in block 'header'
May 16 14:17:07 mw116-m python[8309]:         {% for exporter in get_frontend_exporters() %}
May 16 14:17:07 mw116-m python[8309]:       File "/opt/conda/miniconda3/lib/python3.10/site-packages/notebook/notebook/handlers.py", line 23, in get_frontend_exporters
May 16 14:17:07 mw116-m python[8309]:         from nbconvert.exporters.base import get_export_names, get_exporter
May 16 14:17:07 mw116-m python[8309]:       File "/opt/conda/miniconda3/lib/python3.10/site-packages/nbconvert/__init__.py", line 4, in <module>
May 16 14:17:07 mw116-m python[8309]:         from .exporters import *
May 16 14:17:07 mw116-m python[8309]:       File "/opt/conda/miniconda3/lib/python3.10/site-packages/nbconvert/exporters/__init__.py", line 3, in <module>
May 16 14:17:07 mw116-m python[8309]:         from .html import HTMLExporter
May 16 14:17:07 mw116-m python[8309]:       File "/opt/conda/miniconda3/lib/python3.10/site-packages/nbconvert/exporters/html.py", line 12, in <module>
May 16 14:17:07 mw116-m python[8309]:         from jinja2 import contextfilter
May 16 14:17:07 mw116-m python[8309]:     ImportError: cannot import name 'contextfilter' from 'jinja2' (/opt/conda/miniconda3/lib/python3.10/site-packages/jinja2/__init__.py)
May 16 14:17:07 mw116-m python[8309]: [D 14:17:07.045 NotebookApp] Using contents: services/contents
May 16 14:17:07 mw116-m python[8309]: [D 14:17:07.045 NotebookApp] Using contents: services/contents
May 16 14:17:07 mw116-m python[8309]: [E 14:17:07.046 NotebookApp] {
May 16 14:17:07 mw116-m python[8309]:       "Host": "localhost:8123",
May 16 14:17:07 mw116-m python[8309]:       "Connection": "keep-alive",
May 16 14:17:07 mw116-m python[8309]:       "Sec-Ch-Ua": "\"Google Chrome\";v=\"113\", \"Chromium\";v=\"113\", \"Not-A.Brand\";v=\"24\"",
May 16 14:17:07 mw116-m python[8309]:       "Sec-Ch-Ua-Mobile": "?0",
May 16 14:17:07 mw116-m python[8309]:       "Sec-Ch-Ua-Platform": "\"macOS\"",
May 16 14:17:07 mw116-m python[8309]:       "Upgrade-Insecure-Requests": "1",
May 16 14:17:07 mw116-m python[8309]:       "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36",
May 16 14:17:07 mw116-m python[8309]:       "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7",
May 16 14:17:07 mw116-m python[8309]:       "Sec-Fetch-Site": "same-origin",
May 16 14:17:07 mw116-m python[8309]:       "Sec-Fetch-Mode": "navigate",
May 16 14:17:07 mw116-m python[8309]:       "Sec-Fetch-User": "?1",
May 16 14:17:07 mw116-m python[8309]:       "Sec-Fetch-Dest": "document",
May 16 14:17:07 mw116-m python[8309]:       "Referer": "http://localhost:8123/tree/gnomad-mwilson/v4",
May 16 14:17:07 mw116-m python[8309]:       "Accept-Encoding": "gzip, deflate, br",
May 16 14:17:07 mw116-m python[8309]:       "Accept-Language": "en-US,en;q=0.9",
May 16 14:17:07 mw116-m python[8309]:       "Cookie": "_xsrf=2|9545df9a|e2e3b7e28eb05f2672217e412f31369d|1684246458"
May 16 14:17:07 mw116-m python[8309]:     }

notebook.log

Version

0.2.116

Relevant log output

No response