404 errors when creating or opening notebooks in jupyterlab 4 under reverse proxy

[andrewssobral]

Description

I am able to login, the Terminal works fine, but when I try to create/open a Notebook i got "File Load Error for Untitled.ipynb Invalid response: 404 Not Found"

Reproduce

FROM activeeon/python:3.12

WORKDIR /opt/notebooks

ENV DEBIAN_FRONTEND="noninteractive"
RUN apt update && apt upgrade -y

# Install required packages
# ------------------------
RUN apt-get install -y \
        apt-utils \
        nodejs \
        gcc \
        g++ \
        make \
        yarn \
        manpages-dev \
        graphviz \
        libgraphviz-dev

# Install python libraries
# ------------------------
RUN python3 -m pip install --upgrade pip
RUN python3 -m pip install --pre proactive
RUN python3 -m pip install --index-url https://download.pytorch.org/whl/cpu --extra-index-url https://pypi.org/simple jupyterlab==4.1.4 notebook==7.1.1 beakerx==2.4.1 scikit-learn==1.4.1.post1 pandas==2.2.1 tensorflow==2.16.1 keras==3.0.5 tensorboard==2.16.2 torch torchvision torchaudio
RUN python3 -m pip install ipython==8.22.2

RUN python3 -m pip install "git+https://github.com/ow2-proactive/proactive-jupyter-kernel.git@jupyterlab4"
RUN python3 -m proactive-jupyter-kernel.install

# Clean up
# --------
RUN python3 -m pip cache purge
RUN rm -rf ~/.cache/pip
RUN apt-get autoclean && apt-get clean \
        && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*

# Expose JupyterLab
# -----------------
EXPOSE 8888

ENV JUPYTERLAB_PASSWORD=''
ENV ADDITIONAL_OPTIONS=''

# ENTRYPOINT
COPY entrypoint.sh /usr/local/bin/entrypoint.sh
RUN chmod +x /usr/local/bin/entrypoint.sh
ENTRYPOINT ["/usr/local/bin/entrypoint.sh"]

#!/bin/bash

# Generate the hashed password
HASHED_PASSWORD=$(python3 -c "from jupyter_server.auth import passwd; print(passwd(passphrase='$JUPYTERLAB_PASSWORD', algorithm='sha1'))")

# Execute JupyterLab with additional options
exec jupyter lab --port=8888 --ip=0.0.0.0 --no-browser --allow-root --ServerApp.password="$HASHED_PASSWORD" $ADDITIONAL_OPTIONS

docker run -it --rm --name jupyterlab-server-684 -p 43751:8888 -e ADDITIONAL_OPTIONS="--ServerApp.root_dir=/opt/notebooks --ServerApp.allow_remote_access=True --ServerApp.trust_xheaders=True --ServerApp.disable_check_xsrf=True --ServerApp.websocket_url='wss://try.activeeon.com:8443/cloud-automation-service/services/12/endpoints/jupyterlab-endpoint-684/' --ServerApp.allow_origin='*' " activeeon/jupyterlab4:latest

No errors in the logs:

$ docker run -it --rm --name jupyterlab-server-684 -p 43751:8888 -e ADDITIONAL_OPTIONS="--ServerApp.root_dir=/opt/notebooks --ServerApp.allow_remote_access=True --ServerApp.trust_xheaders=True --ServerApp.disable_check_xsrf=True --ServerApp.websocket_url='wss://try.activeeon.com:8443/cloud-automation-service/services/12/endpoints/jupyterlab-endpoint-684/' --ServerApp.allow_origin='*' --ServerApp.custom_display_url='https://try.activeeon.com:8443/cloud-automation-service/services/12/endpoints/jupyterlab-endpoint-684/lab' " activeeon/jupyterlab4:latest
[W 2024-03-13 16:12:51.652 ServerApp] ServerApp.password config is deprecated in 2.0. Use PasswordIdentityProvider.hashed_password.
[I 2024-03-13 16:12:52.448 ServerApp] beakerx | extension was successfully linked.
[I 2024-03-13 16:12:52.448 ServerApp] jupyter_lsp | extension was successfully linked.
[I 2024-03-13 16:12:52.453 ServerApp] jupyter_server_terminals | extension was successfully linked.
[I 2024-03-13 16:12:52.458 ServerApp] jupyterlab | extension was successfully linked.
[I 2024-03-13 16:12:52.463 ServerApp] notebook | extension was successfully linked.
[I 2024-03-13 16:12:52.464 ServerApp] Writing Jupyter server cookie secret to /root/.local/share/jupyter/runtime/jupyter_cookie_secret
[I 2024-03-13 16:12:52.467 ServerApp] notebook_shim | extension was successfully linked.
[I 2024-03-13 16:12:52.482 ServerApp] notebook_shim | extension was successfully loaded.
[I 2024-03-13 16:12:52.484 ServerApp] Registered beakerx server extension at URL path /beakerx
[I 2024-03-13 16:12:52.484 ServerApp] beakerx | extension was successfully loaded.
[I 2024-03-13 16:12:52.487 ServerApp] jupyter_lsp | extension was successfully loaded.
[I 2024-03-13 16:12:52.488 ServerApp] jupyter_server_terminals | extension was successfully loaded.
[I 2024-03-13 16:12:52.489 LabApp] JupyterLab extension loaded from /usr/local/lib/python3.12/site-packages/jupyterlab
[I 2024-03-13 16:12:52.489 LabApp] JupyterLab application directory is /usr/local/share/jupyter/lab
[I 2024-03-13 16:12:52.490 LabApp] Extension Manager is 'pypi'.
[I 2024-03-13 16:12:52.520 ServerApp] jupyterlab | extension was successfully loaded.
[I 2024-03-13 16:12:52.524 ServerApp] notebook | extension was successfully loaded.
[I 2024-03-13 16:12:52.524 ServerApp] Serving notebooks from local directory: /opt/notebooks
[I 2024-03-13 16:12:52.524 ServerApp] Jupyter Server 2.13.0 is running at:
[I 2024-03-13 16:12:52.524 ServerApp] https://try.activeeon.com:8443/cloud-automation-service/services/12/endpoints/jupyterlab-endpoint-684/lab
[I 2024-03-13 16:12:52.524 ServerApp]     http://127.0.0.1:8888/lab
[I 2024-03-13 16:12:52.524 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[I 2024-03-13 16:12:52.547 ServerApp] Skipped non-installed server(s): bash-language-server, dockerfile-language-server-nodejs, javascript-typescript-langserver, jedi-language-server, julia-language-server, pyright, python-language-server, python-lsp-server, r-languageserver, sql-language-server, texlab, typescript-language-server, unified-language-server, vscode-css-languageserver-bin, vscode-html-languageserver-bin, vscode-json-languageserver-bin, yaml-language-server
[I 2024-03-13 16:12:56.602 ServerApp] 302 GET / (@78.199.116.24) 0.37ms
[I 2024-03-13 16:12:56.702 LabApp] 302 GET /lab (@78.199.116.24) 3.03ms
[I 2024-03-13 16:12:59.890 ServerApp] User cfe6181568b647e8a04105e3131d89cd logged in.
[I 2024-03-13 16:12:59.890 ServerApp] 302 POST /login?next=%2Flab (cfe6181568b647e8a04105e3131d89cd@78.199.116.24) 3.37ms
[I 2024-03-13 16:13:01.889 LabApp] Build is up to date
[I 2024-03-13 16:13:05.443 ServerApp] New terminal with automatic name: 1
[I 2024-03-13 16:13:12.692 ServerApp] EOF on FD 25; stopping reading
[I 2024-03-13 16:13:12.793 ServerApp] Terminal 1 closed
[I 2024-03-13 16:13:15.317 ServerApp] Creating new notebook in 
[I 2024-03-13 16:13:15.373 ServerApp] Writing notebook-signing key to /root/.local/share/jupyter/notebook_secret
[I 2024-03-13 16:13:51.865 ServerApp] 302 GET / (@78.199.116.24) 1.24ms
[I 2024-03-13 16:13:53.832 LabApp] Build is up to date
[I 2024-03-13 16:14:23.137 LabApp] Build is up to date
[I 2024-03-13 16:14:30.307 ServerApp] New terminal with automatic name: 1
[I 2024-03-13 16:14:36.589 ServerApp] EOF on FD 28; stopping reading
[I 2024-03-13 16:14:36.690 ServerApp] Terminal 1 closed
[I 2024-03-13 16:14:41.776 ServerApp] Creating new notebook in 
[I 2024-03-13 16:18:57.789 ServerApp] New terminal with automatic name: 1
[I 2024-03-13 16:19:14.213 ServerApp] Creating new notebook in

This is the login page:

Note that the URL is correct: https://try.activeeon.com:8443/cloud-automation-service/services/12/endpoints/jupyterlab-endpoint-684/login?next=%2Flab

After login, I am here:

Note that the URL is wrong: https://try.activeeon.com:8443/lab

The Terminal works fine:

When I try to create/open a Notebook i got the following error message

More details

I tried to add those infos (separately and together) just after --ServerApp.allow_origin='*' in the base command below, but i am still getting the same error:

Tested options:
--ServerApp.base_url='/cloud-automation-service/services/12/endpoints/jupyterlab-endpoint-684/'
--ServerApp.default_url='/cloud-automation-service/services/12/endpoints/jupyterlab-endpoint-684/'
--ServerApp.custom_display_url='https://try.activeeon.com:8443/cloud-automation-service/services/12/endpoints/jupyterlab-endpoint-684/lab'

Base command:
docker run -it --rm --name jupyterlab-server-684 -p 43751:8888 -e ADDITIONAL_OPTIONS="--ServerApp.root_dir=/opt/notebooks --ServerApp.allow_remote_access=True --ServerApp.trust_xheaders=True --ServerApp.disable_check_xsrf=True --ServerApp.websocket_url='wss://try.activeeon.com:8443/cloud-automation-service/services/12/endpoints/jupyterlab-endpoint-684/' --ServerApp.allow_origin='*' " activeeon/jupyterlab4:latest

Someone knows how can I fix this issue?

[welcome]

Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! 🤗

If you haven't done so already, check out Jupyter's Code of Conduct. Also, please try to follow the issue template as it helps other other community members to contribute more effectively.

You can meet the other Jovyans by joining our Discourse forum. There is also an intro thread there where you can stop by and say Hi! 👋

Welcome to the Jupyter community! 🎉

[andrewssobral]

For info, my procedure works fine with the JupyterLab 2 and 3, but not 4

[andrewssobral]

I think the problem comes from the following request:
https://try.activeeon.com:8443/api/contents/Untitled1.ipynb?type=notebook&content=1&hash=1&1710346479189
instead of:
https://try.activeeon.com:8443/cloud-automation-service/services/12/endpoints/jupyterlab-endpoint-684/api/contents/Untitled1.ipynb?type=notebook&content=1&hash=1&1710346479189
but those options not worked for me:
--ServerApp.base_url='/cloud-automation-service/services/12/endpoints/jupyterlab-endpoint-684/'
--ServerApp.default_url='/cloud-automation-service/services/12/endpoints/jupyterlab-endpoint-684/'
--ServerApp.custom_display_url='https://try.activeeon.com:8443/cloud-automation-service/services/12/endpoints/jupyterlab-endpoint-684/lab'

[JasonWeill]

@krassowski shared #11393 and #7966, two older issues concerning using JupyterLab with a reverse proxy.