Zombie proxy configurations

[mskblackbelt]

Bug description

When I try to set base_url to /jupyter and run sudo tljh-config reload proxy, navigation to domain.com/jupyter redirects to domain.com/hub/jupyter and throws a 404 error.

Expected behaviour

Navigation to domain.com/jupyter should load the JupyterHub (either login screen or logged in user's instance). Navigation to domain.com should redirect to domain.com/jupyter.

Actual behaviour

Navigation to domain.com redirects to domain.com/hub/jupyter, and JupyterHub gives a 404 error.

How to reproduce

• Run sudo tljh-config set c.JupyterHub.base_url "/jupyter"
• Run sudo tljh-config reload for hub and proxy (or sudo systemctl restart jupyterhub traefik) to reload settings
• Navigate to domain.com
• Get 404 page

I also tried to undo this by running sudo tljh-config unset c.JupyterHub.base_url and again reloading the hub and proxy, expecting this to make the issue disappear. However, it seems that there's some zombie configuration file for Traefik that I can't clear. Even though base_url is no longer present in tljh-config, my hub is still redirecting URLs to /hub/jupyter and giving 404 errors.

I also tried setting some variable in the custom TOML files under /opt/tljh/config/traefik_config.d, and those settings also won't go away (I removed the custom file from the folder, but settings from that file persist across service and server restarts). This leads me to suspect the issue is related to a configuration file for Traefik that's hiding somewhere, but I don't know where to look for this.

Your personal set up

TLJH installed using script curl-ed on 2021-01-20 using default settings + NativeAuthenticator

• OS:

Ubuntu 18.04

• Version(s):

jupyterhub: 1.2.2
python: 3.7.9 (TLJH), 3.6.9 (system)

• Full environment

# packages in environment at /opt/tljh/user:
#
# Name                    Version                   Build  Channel
_anaconda_depends         2020.07                  py37_0  
_libgcc_mutex             0.1                        main  
alabaster                 0.7.12                   py37_0  
alembic                   1.5.2                    pypi_0    pypi
anaconda                  custom                   py37_1  
anaconda-client           1.7.2                    py37_0  
anaconda-project          0.9.1              pyhd3eb1b0_1  
anyio                     2.0.2            py37h89c1867_4    conda-forge
appdirs                   1.4.4                      py_0  
argh                      0.26.2                   py37_0  
argon2-cffi               20.1.0           py37h7b6447c_1  
asn1crypto                1.4.0                      py_0  
astroid                   2.4.2                    py37_0  
astropy                   4.2              py37h27cfd23_0  
async_generator           1.10             py37h28b3542_0  
atomicwrites              1.4.0                      py_0  
attrs                     20.3.0             pyhd3eb1b0_0  
autopep8                  1.5.4                      py_0  
avogadro                  1.92.1                   pypi_0    pypi
babel                     2.9.0              pyhd3eb1b0_0  
backcall                  0.2.0                      py_0  
backports                 1.0                pyhd3eb1b0_2  
backports.shutil_get_terminal_size 1.0.0              pyhd3eb1b0_3  
beautifulsoup4            4.9.3              pyha847dfd_0  
biopython                 1.78             py37h7b6447c_0  
bitarray                  1.6.1            py37h27cfd23_0  
bkcharts                  0.2                      py37_0  
black                     19.10b0                    py_0  
blas                      1.0                         mkl  
bleach                    3.2.2                    pypi_0    pypi
blosc                     1.20.1               hd408876_0  
bokeh                     2.2.3                    py37_0  
boto                      2.49.0                   py37_0  
bottleneck                1.3.2            py37heb32a55_1  
brotli                    1.0.9                he6710b0_2  
brotlipy                  0.7.0           py37h27cfd23_1003  
brunsli                   0.1                  h2531618_0  
bzip2                     1.0.8                h7b6447c_0  
ca-certificates           2021.1.19            h06a4308_0  
cairo                     1.16.0            h18b612c_1001    conda-forge
cclib                     1.7                pyhd3deb0d_0    conda-forge
certifi                   2020.12.5        py37h06a4308_0  
certipy                   0.1.3                    pypi_0    pypi
cffi                      1.14.4           py37h261ae71_0  
chardet                   4.0.0           py37h06a4308_1003  
charls                    2.1.0                he6710b0_2  
click                     7.1.2              pyhd3eb1b0_0  
cloudpickle               1.6.0                      py_0  
clyent                    1.2.2                    py37_1  
colorama                  0.4.4              pyhd3eb1b0_0  
conda                     4.9.2            py37h06a4308_0  
conda-package-handling    1.7.2            py37h03888b9_0  
contextlib2               0.6.0.post1                py_0  
cryptography              3.3.1            py37h3c74f83_0  
curl                      7.71.1               hbc83047_1  
cycler                    0.10.0                   py37_0  
cython                    0.29.21          py37h2531618_0  
cytoolz                   0.11.0           py37h7b6447c_0  
dask                      2020.12.0          pyhd3eb1b0_0  
dask-core                 2020.12.0          pyhd3eb1b0_0  
dbus                      1.13.18              hb2f20db_0  
decorator                 4.4.2                      py_0  
defusedxml                0.6.0                    pypi_0    pypi
diff-match-patch          20200713                   py_0  
diskcache                 5.1.0                    pypi_0    pypi
distributed               2021.1.0         py37h06a4308_0  
docutils                  0.16                     py37_1  
entrypoints               0.3                      pypi_0    pypi
et_xmlfile                1.0.1                   py_1001  
expat                     2.2.10               he6710b0_2  
fastcache                 1.1.0            py37h7b6447c_0  
filelock                  3.0.12             pyhd3eb1b0_1  
flake8                    3.8.4                      py_0  
flask                     1.1.2                      py_0  
fontconfig                2.13.1            he4413a7_1000    conda-forge
freetype                  2.10.4               h5ab3b9f_0  
fribidi                   1.0.10               h7b6447c_0  
fsspec                    0.8.3                      py_0  
future                    0.18.2                   py37_1  
get_terminal_size         1.0.0                haa9412d_0  
gevent                    21.1.1           py37h27cfd23_1  
giflib                    5.1.4                h14c3975_1  
girder-client             3.1.3                    pypi_0    pypi
glib                      2.66.1               h92f7085_0  
glob2                     0.7                pyhd3eb1b0_0  
gmp                       6.1.2                h6c8ec71_1  
gmpy2                     2.0.8            py37h10f8cd9_2  
graphite2                 1.3.14               h23475e2_0  
greenlet                  0.4.17           py37h7b6447c_0  
gst-plugins-base          1.14.0               h8213a91_2  
gstreamer                 1.14.0               h28cd5cc_2  
h5py                      2.10.0           py37h7918eee_0  
harfbuzz                  2.4.0                hca77d97_1  
hdf5                      1.10.4               hb1b8bf9_0  
heapdict                  1.0.1                      py_0  
html5lib                  1.1                        py_0  
icu                       58.2                 he6710b0_3  
idna                      2.10               pyhd3eb1b0_0  
imagecodecs               2021.1.11        py37h581e88b_1  
imageio                   2.9.0                      py_0  
imagesize                 1.2.0                      py_0  
importlib-metadata        3.4.0                    pypi_0    pypi
importlib_metadata        2.0.0                         1  
iniconfig                 1.1.1              pyhd3eb1b0_0  
intel-openmp              2020.2                      254  
intervaltree              3.1.0                      py_0  
ipykernel                 5.4.3                    pypi_0    pypi
ipympl                    0.6.3              pyhd8ed1ab_0    conda-forge
ipython                   7.19.0           py37hb070fc8_1  
ipython_genutils          0.2.0              pyhd3eb1b0_1  
ipywidgets                7.5.1                    pypi_0    pypi
isort                     5.6.4                      py_0  
itsdangerous              1.1.0                    py37_0  
jbig                      2.1                  hdba287a_0  
jdcal                     1.4.1                      py_0  
jedi                      0.17.2           py37h06a4308_1  
jeepney                   0.6.0              pyhd3eb1b0_0  
jinja2                    2.11.2             pyhd3eb1b0_0  
joblib                    1.0.0              pyhd3eb1b0_0  
jpeg                      9b                   h024ee3a_2  
json5                     0.9.5                      py_0  
jsonpath-rw               1.4.0                    pypi_0    pypi
jsonschema                3.2.0                      py_2  
jupyter                   1.0.0                    py37_7  
jupyter-client            6.1.11                   pypi_0    pypi
jupyter-telemetry         0.1.0                    pypi_0    pypi
jupyter_client            6.1.7                      py_0  
jupyter_console           6.2.0                      py_0  
jupyter_core              4.7.0            py37h06a4308_0  
jupyter_server            1.2.2            py37h89c1867_1    conda-forge
jupyterhub                1.2.2                    pypi_0    pypi
jupyterlab                2.2.9                    pypi_0    pypi
jupyterlab-templates      0.3.0                    pypi_0    pypi
jupyterlab_pygments       0.1.2                      py_0  
jupyterlab_server         2.1.2              pyhd8ed1ab_0    conda-forge
jupyterlab_widgets        1.0.0              pyhd3eb1b0_1  
jupytext                  1.9.1                    pypi_0    pypi
jxrlib                    1.1                  h7b6447c_2  
keyring                   21.8.0           py37h06a4308_0  
kiwisolver                1.3.0            py37h2531618_0  
krb5                      1.18.2               h173b8e3_0  
lazy-object-proxy         1.4.3            py37h27cfd23_2  
lcms2                     2.11                 h396b838_0  
ld_impl_linux-64          2.33.1               h53a641e_7  
lerc                      2.2.1                h2531618_0  
libaec                    1.0.4                he6710b0_1  
libarchive                3.4.2                h62408e4_0  
libcurl                   7.71.1               h20c2e04_1  
libdeflate                1.7                  h27cfd23_5  
libedit                   3.1.20191231         h14c3975_1  
libffi                    3.3                  he6710b0_2  
libgcc-ng                 9.1.0                hdf63c60_0  
libgfortran-ng            7.3.0                hdf63c60_0  
liblief                   0.10.1               he6710b0_0  
libllvm10                 10.0.1               hbcb73fb_5  
libllvm9                  9.0.1                h4a3c616_1  
libpng                    1.6.37               hbc83047_0  
libsodium                 1.0.18               h7b6447c_0  
libspatialindex           1.9.3                he6710b0_0  
libssh2                   1.9.0                h1ba5d50_1  
libstdcxx-ng              9.1.0                hdf63c60_0  
libtiff                   4.1.0                h2733197_1  
libtool                   2.4.6             h7b6447c_1005  
libuuid                   2.32.1            h14c3975_1000    conda-forge
libwebp                   1.0.1                h8e7db2f_0  
libxcb                    1.14                 h7b6447c_0  
libxml2                   2.9.10               hb55368b_3  
libxslt                   1.1.34               hc22bd24_0  
libzopfli                 1.0.3                he6710b0_0  
llvmlite                  0.34.0           py37h269e1b5_4  
locket                    0.2.1            py37h06a4308_1  
lxml                      4.6.2            py37h9120a33_0  
lz4-c                     1.9.3                h2531618_0  
lzo                       2.10                 h7b6447c_2  
mako                      1.1.4                    pypi_0    pypi
markdown-it-py            0.6.1                    pypi_0    pypi
markupsafe                1.1.1            py37h14c3975_1  
matplotlib                3.3.2                h06a4308_0  
matplotlib-base           3.3.2            py37h817c723_0  
mccabe                    0.6.1                    py37_1  
mdit-py-plugins           0.2.4                    pypi_0    pypi
mistune                   0.8.4           py37h14c3975_1001  
mkl                       2020.2                      256  
mkl-service               2.3.0            py37he8ac12f_0  
mkl_fft                   1.2.0            py37h23d657b_0  
mkl_random                1.1.1            py37h0573a6f_0  
mock                      4.0.3              pyhd3eb1b0_0  
more-itertools            8.6.0              pyhd3eb1b0_0  
mpc                       1.1.0                h10f8cd9_1  
mpfr                      4.0.2                hb69a4c5_1  
mpmath                    1.1.0                    py37_0  
msgpack-python            1.0.1            py37hff7bd54_0  
multipledispatch          0.6.0                    py37_0  
mypy_extensions           0.4.3                    py37_0  
nbclassic                 0.2.6              pyhd8ed1ab_0    conda-forge
nbclient                  0.5.1                      py_0  
nbconvert                 6.0.7                    py37_0  
nbformat                  5.1.2              pyhd3eb1b0_1  
nbgitpuller               0.9.0                      py_0    conda-forge
nbresuse                  0.3.6                    pypi_0    pypi
ncurses                   6.2                  he6710b0_1  
nest-asyncio              1.4.3              pyhd3eb1b0_0  
networkx                  2.5                        py_0  
nltk                      3.5                        py_0  
nmrglue                   0.8              py37hf01694f_0    bioconda
nose                      1.3.7           pyhd3eb1b0_1006  
notebook                  6.1.5                    pypi_0    pypi
nteract-on-jupyter        2.1.3                    pypi_0    pypi
numba                     0.51.2           py37h04863e7_1  
numexpr                   2.7.2            py37hb2eb853_0  
numpy                     1.19.2           py37h54aff64_0  
numpy-base                1.19.2           py37hfa32c7d_0  
numpydoc                  1.1.0              pyhd3eb1b0_1  
oauthlib                  3.1.0                    pypi_0    pypi
olefile                   0.46                     py37_0  
openbabel                 3.1.1            py37h200e996_1    conda-forge
openchemistry             0.0.20                   pypi_0    pypi
openjpeg                  2.3.0                h05c96fa_1  
openpyxl                  3.0.6              pyhd3eb1b0_0  
openssl                   1.1.1i               h27cfd23_0  
packaging                 20.8               pyhd3eb1b0_0  
pamela                    1.0.0                    pypi_0    pypi
pandas                    1.2.1            py37ha9443f7_0  
pandoc                    2.11                 hb0f4dca_0  
pandocfilters             1.4.3            py37h06a4308_1  
pango                     1.45.3               hd140c19_0  
parso                     0.7.1                    pypi_0    pypi
partd                     1.1.0                      py_0  
patchelf                  0.12                 h2531618_1  
path                      15.0.1           py37h06a4308_0  
path.py                   12.5.0                        0  
pathlib2                  2.3.5            py37h06a4308_2  
pathspec                  0.7.0                      py_0  
pathtools                 0.1.2                      py_1  
patsy                     0.5.1                    py37_0  
pcre                      8.44                 he6710b0_0  
pep8                      1.7.1                    py37_0  
periodictable             1.5.2                      py_0    conda-forge
pexpect                   4.8.0              pyhd3eb1b0_3  
pickleshare               0.7.5           pyhd3eb1b0_1003  
pillow                    8.1.0            py37he98fc37_0  
pip                       20.3.3           py37h06a4308_0  
pixman                    0.38.0            h516909a_1003    conda-forge
pkginfo                   1.7.0            py37h06a4308_0  
pluggy                    0.13.1                   py37_0  
ply                       3.11                     py37_0  
prometheus_client         0.9.0              pyhd3eb1b0_0  
prompt-toolkit            3.0.11                   pypi_0    pypi
prompt_toolkit            3.0.8                         0  
psutil                    5.8.0                    pypi_0    pypi
ptyprocess                0.7.0              pyhd3eb1b0_2  
py                        1.10.0             pyhd3eb1b0_0  
py-lief                   0.10.1           py37h403a769_0  
pycodestyle               2.6.0                      py_0  
pycosat                   0.6.3            py37h27cfd23_0  
pycparser                 2.20                       py_2  
pycrypto                  2.6.1           py37h7b6447c_10  
pycurl                    7.43.0.6         py37h1ba5d50_0  
pydocstyle                5.1.1                      py_0  
pyerfa                    1.7.1.1          py37h27cfd23_1  
pyflakes                  2.2.0                      py_0  
pygments                  2.7.4              pyhd3eb1b0_0  
pylint                    2.6.0                    py37_0  
pyls-black                0.4.6                hd3eb1b0_0  
pyls-spyder               0.3.0              pyhd3eb1b0_0  
pyodbc                    4.0.30           py37he6710b0_0  
pyopenssl                 20.0.1             pyhd3eb1b0_1  
pyparsing                 2.4.7              pyhd3eb1b0_0  
pyqt                      5.9.2            py37h05f1152_2  
pyrsistent                0.17.3           py37h7b6447c_0  
pysocks                   1.7.1                    py37_1  
pytables                  3.6.1            py37h71ec239_0  
pytest                    6.2.1            py37h06a4308_0  
python                    3.7.9                h7579374_0  
python-dateutil           2.8.1                      py_0  
python-editor             1.0.4                    pypi_0    pypi
python-json-logger        2.0.1                    pypi_0    pypi
python-jsonrpc-server     0.4.0                      py_0  
python-language-server    0.36.2             pyhd3eb1b0_0  
python-libarchive-c       2.9                pyhd3eb1b0_0  
python_abi                3.7                     1_cp37m    conda-forge
pytz                      2020.5             pyhd3eb1b0_0  
pywavelets                1.1.1            py37h7b6447c_2  
pyxdg                     0.27               pyhd3eb1b0_0  
pyyaml                    5.4              py37h27cfd23_1  
pyzmq                     21.0.1                   pypi_0    pypi
qdarkstyle                2.8.1                      py_0  
qt                        5.9.7                h5867ecd_1  
qtawesome                 1.0.1                      py_0  
qtconsole                 4.7.7                      py_0  
qtpy                      1.9.0                      py_0  
readline                  8.0                  h7b6447c_0  
regex                     2020.11.13       py37h27cfd23_0  
requests                  2.25.1             pyhd3eb1b0_0  
requests-toolbelt         0.9.1                    pypi_0    pypi
ripgrep                   12.1.1                        0  
rmsd                      1.4                      pypi_0    pypi
rope                      0.18.0                     py_0  
rtree                     0.9.4                    py37_1  
ruamel-yaml               0.16.12                  pypi_0    pypi
ruamel-yaml-clib          0.2.2                    pypi_0    pypi
ruamel_yaml               0.15.87          py37h7b6447c_1  
scikit-image              0.17.2           py37hdf5156a_0  
scikit-learn              0.23.2           py37h0573a6f_0  
scipy                     1.5.2            py37h0b6359f_0  
seaborn                   0.11.1             pyhd3eb1b0_0  
secretstorage             3.3.0            py37h06a4308_0  
send2trash                1.5.0              pyhd3eb1b0_1  
setuptools                51.3.3           py37h06a4308_4  
simplegeneric             0.8.1                    py37_2  
singledispatch            3.4.0.3                 py_1001  
sip                       4.19.8           py37hf484d3e_0  
six                       1.15.0           py37h06a4308_0  
snappy                    1.1.8                he6710b0_0  
sniffio                   1.1.0                    py37_2  
snowballstemmer           2.0.0                      py_0  
sortedcollections         2.1.0              pyhd3eb1b0_0  
sortedcontainers          2.3.0              pyhd3eb1b0_0  
soupsieve                 2.1                pyhd3eb1b0_0  
sphinx                    3.4.3              pyhd3eb1b0_0  
sphinxcontrib             1.0                      py37_1  
sphinxcontrib-applehelp   1.0.2                      py_0  
sphinxcontrib-devhelp     1.0.2                      py_0  
sphinxcontrib-htmlhelp    1.0.3                      py_0  
sphinxcontrib-jsmath      1.0.1                      py_0  
sphinxcontrib-qthelp      1.0.3                      py_0  
sphinxcontrib-serializinghtml 1.1.4                      py_0  
sphinxcontrib-websupport  1.2.4                      py_0  
spyder                    4.2.0            py37h06a4308_0  
spyder-kernels            1.10.1           py37h06a4308_0  
sqlalchemy                1.3.22                   pypi_0    pypi
sqlite                    3.33.0               h62c20be_0  
statsmodels               0.12.1           py37h27cfd23_0  
symfit                    0.5.3            py37h89c1867_1    conda-forge
sympy                     1.7.1            py37h06a4308_0  
tbb                       2020.3               hfd86e86_0  
tblib                     1.7.0                      py_0  
terminado                 0.9.2                    pypi_0    pypi
testpath                  0.4.4                      py_0  
threadpoolctl             2.1.0              pyh5ca1d4c_0  
three-merge               0.1.1              pyhd3eb1b0_0  
tifffile                  2021.1.14          pyhd3eb1b0_1  
tk                        8.6.10               hbc83047_0  
toml                      0.10.1                     py_0  
toolz                     0.11.1             pyhd3eb1b0_0  
toposort                  1.6                pyhd8ed1ab_0    conda-forge
tornado                   5.1.1                    pypi_0    pypi
tqdm                      4.55.1             pyhd3eb1b0_0  
traitlets                 5.0.5                      py_0  
typed-ast                 1.4.2            py37h27cfd23_1  
typing_extensions         3.7.4.3                    py_0  
ujson                     4.0.1            py37he6710b0_0  
unicodecsv                0.14.1                   py37_0  
unixodbc                  2.3.9                h7b6447c_0  
urllib3                   1.26.2             pyhd3eb1b0_0  
watchdog                  0.10.4           py37h06a4308_0  
wcwidth                   0.2.5                      py_0  
webencodings              0.5.1                    pypi_0    pypi
werkzeug                  1.0.1                      py_0  
wheel                     0.36.2             pyhd3eb1b0_0  
widgetsnbextension        3.5.1                    py37_0  
wrapt                     1.11.2           py37h7b6447c_0  
wurlitzer                 2.0.1                    py37_0  
xarray                    0.16.2             pyhd8ed1ab_0    conda-forge
xlrd                      2.0.1              pyhd3eb1b0_0  
xlsxwriter                1.3.7                      py_0  
xlwt                      1.3.0                    py37_0  
xorg-kbproto              1.0.7             h14c3975_1002    conda-forge
xorg-libice               1.0.10               h516909a_0    conda-forge
xorg-libsm                1.2.3             h84519dc_1000    conda-forge
xorg-libx11               1.6.12               h516909a_0    conda-forge
xorg-libxext              1.3.4                h516909a_0    conda-forge
xorg-libxrender           0.9.10            h516909a_1002    conda-forge
xorg-renderproto          0.11.1            h14c3975_1002    conda-forge
xorg-xextproto            7.3.0             h14c3975_1002    conda-forge
xorg-xproto               7.0.31            h27cfd23_1007  
xz                        5.2.5                h7b6447c_0  
yaml                      0.2.5                h7b6447c_0  
yapf                      0.30.0                     py_0  
zeromq                    4.3.3                he6710b0_3  
zfp                       0.5.5                h2531618_4  
zict                      2.0.0                      py_0  
zipp                      3.4.0              pyhd3eb1b0_0  
zlib                      1.2.11               h7b6447c_3  
zope                      1.0                      py37_1  
zope.event                4.5.0                    py37_0  
zope.interface            5.2.0            py37h27cfd23_0  
zstd                      1.4.5                h9ceee32_0 

• Configuration

# tljh-config
users:
  admin:
  - dwheeler
user_environment:
  default_app: jupyterlab
limits:
  memory: 3G
  cpu: 4
auth:
  type: nativeauthenticator.NativeAuthenticator
c:
  JupyterLabTemplates:
    template_dirs: /opt/tljh/user/share/jupyter/notebook_templates
    include_default: false
  Authenticator:
    check_common_password: true
  JupyterHub:
    base_url: /jupyter
https:
  enabled: true
  tls:
    key: /etc/ssl/private/sugarcube.key
    cert: /etc/ssl/certs/sugarcube.pem
services:
  cull:
    timeout: 1800

# jupyterhub_config.py
c.JupyterLabTemplates.include_default = False

• Logs

Jan 28 11:58:53 sugarcube python3[8553]: [W 2021-01-28 11:58:53.759 JupyterHub log:181] 404 GET /hub/jupyter/ (test@***.***.***.***) 19.27ms
Jan 28 11:58:53 sugarcube python3[8553]: [I 2021-01-28 11:58:53.724 JupyterHub log:181] 302 GET /jupyter/ -> /hub/jupyter/ (@***.***.***.***) 1.62ms
Jan 28 11:58:32 sugarcube python3[8553]: [I 2021-01-28 11:58:32.895 JupyterHub log:181] 200 GET /hub/api/users (cull-idle@127.0.0.1) 27.30ms
Jan 28 11:57:45 sugarcube python3[8553]: [W 2021-01-28 11:57:45.090 JupyterHub log:181] 404 GET /hub/jupyter (test@***.***.***.***) 65.51ms
Jan 28 11:57:44 sugarcube python3[8553]: [I 2021-01-28 11:57:44.991 JupyterHub log:181] 302 GET /jupyter -> /hub/jupyter (@***.***.***.***) 1.62ms
Jan 28 11:57:35 sugarcube python3[8553]: [I 2021-01-28 11:57:35.769 JupyterHub app:2722] JupyterHub is now running at http://:8000
Jan 28 11:57:32 sugarcube python3[8553]: [I 2021-01-28 11:57:32.897 JupyterHub log:181] 200 GET /hub/api/users (cull-idle@127.0.0.1) 30.28ms
Jan 28 11:57:32 sugarcube python3[8553]:     
Jan 28 11:57:32 sugarcube python3[8553]:     tornado.httpclient.HTTPClientError: HTTP 404: Not Found
Jan 28 11:57:32 sugarcube python3[8553]:         validate_cert=self.traefik_api_validate_cert,
Jan 28 11:57:32 sugarcube python3[8553]:       File "/opt/tljh/hub/lib/python3.6/site-packages/jupyterhub_traefik_proxy/proxy.py", line 164, in _traefik_api_request
Jan 28 11:57:32 sugarcube python3[8553]:         resp = await self._traefik_api_request(path)
Jan 28 11:57:32 sugarcube python3[8553]:       File "/opt/tljh/hub/lib/python3.6/site-packages/jupyterhub_traefik_proxy/proxy.py", line 120, in _check_for_traefik_endpoint
Jan 28 11:57:32 sugarcube python3[8553]:     Traceback (most recent call last):
Jan 28 11:57:32 sugarcube python3[8553]: [E 2021-01-28 11:57:32.810 JupyterHub proxy:123] Error checking traefik api for backend /
Jan 28 11:57:32 sugarcube python3[8553]:     
Jan 28 11:57:32 sugarcube python3[8553]:     tornado.httpclient.HTTPClientError: HTTP 404: Not Found
Jan 28 11:57:32 sugarcube python3[8553]:         validate_cert=self.traefik_api_validate_cert,
Jan 28 11:57:32 sugarcube python3[8553]:       File "/opt/tljh/hub/lib/python3.6/site-packages/jupyterhub_traefik_proxy/proxy.py", line 164, in _traefik_api_request
Jan 28 11:57:32 sugarcube python3[8553]:         resp = await self._traefik_api_request(path)
Jan 28 11:57:32 sugarcube python3[8553]:       File "/opt/tljh/hub/lib/python3.6/site-packages/jupyterhub_traefik_proxy/proxy.py", line 120, in _check_for_traefik_endpoint
Jan 28 11:57:32 sugarcube python3[8553]:     Traceback (most recent call last):
Jan 28 11:57:32 sugarcube python3[8553]: [E 2021-01-28 11:57:32.767 JupyterHub proxy:123] Error checking traefik api for backend /user/test
Jan 28 11:57:32 sugarcube python3[8553]:     
Jan 28 11:57:32 sugarcube python3[8553]:     tornado.httpclient.HTTPClientError: HTTP 404: Not Found
Jan 28 11:57:32 sugarcube python3[8553]:         validate_cert=self.traefik_api_validate_cert,
Jan 28 11:57:32 sugarcube python3[8553]:       File "/opt/tljh/hub/lib/python3.6/site-packages/jupyterhub_traefik_proxy/proxy.py", line 164, in _traefik_api_request
Jan 28 11:57:32 sugarcube python3[8553]:         resp = await self._traefik_api_request(path)
Jan 28 11:57:32 sugarcube python3[8553]:       File "/opt/tljh/hub/lib/python3.6/site-packages/jupyterhub_traefik_proxy/proxy.py", line 120, in _check_for_traefik_endpoint
Jan 28 11:57:32 sugarcube python3[8553]:     Traceback (most recent call last):
Jan 28 11:57:32 sugarcube python3[8553]: [E 2021-01-28 11:57:32.741 JupyterHub proxy:123] Error checking traefik api for backend /user/test
Jan 28 11:57:32 sugarcube python3[8553]:     
Jan 28 11:57:32 sugarcube python3[8553]:     tornado.httpclient.HTTPClientError: HTTP 404: Not Found
Jan 28 11:57:32 sugarcube python3[8553]:         validate_cert=self.traefik_api_validate_cert,
Jan 28 11:57:32 sugarcube python3[8553]:       File "/opt/tljh/hub/lib/python3.6/site-packages/jupyterhub_traefik_proxy/proxy.py", line 164, in _traefik_api_request
Jan 28 11:57:32 sugarcube python3[8553]:         resp = await self._traefik_api_request(path)
Jan 28 11:57:32 sugarcube python3[8553]:       File "/opt/tljh/hub/lib/python3.6/site-packages/jupyterhub_traefik_proxy/proxy.py", line 120, in _check_for_traefik_endpoint
Jan 28 11:57:32 sugarcube python3[8553]:     Traceback (most recent call last):
Jan 28 11:57:32 sugarcube python3[8553]: [E 2021-01-28 11:57:32.741 JupyterHub proxy:123] Error checking traefik api for backend /
Jan 28 11:57:32 sugarcube python3[8553]: [I 2021-01-28 11:57:32.739 JupyterHub proxy:135] Waiting for /user/test to register with traefik
Jan 28 11:57:32 sugarcube python3[8553]: [I 2021-01-28 11:57:32.738 JupyterHub proxy:262] Adding user test to proxy /user/test/ => http://127.0.0.1:34803
Jan 28 11:57:32 sugarcube python3[8553]: [I 2021-01-28 11:57:32.738 JupyterHub proxy:135] Waiting for / to register with traefik
Jan 28 11:57:32 sugarcube python3[8553]:   return server_type + "_" + escapism.escape(routespec, safe=safe)
Jan 28 11:57:32 sugarcube python3[8553]: /opt/tljh/hub/lib/python3.6/site-packages/jupyterhub_traefik_proxy/traefik_utils.py:42: RuntimeWarning: Escape character '_' cannot be a safe character. Set allow_collisions=True if you want to allow ambiguous escaped strings.
Jan 28 11:57:32 sugarcube python3[8553]: [W 2021-01-28 11:57:32.737 JupyterHub proxy:345] Adding missing route for /user/test/ (Server(url=http://127.0.0.1:34803/user/test/, bind_url=http://127.0.0.1:34803/user/****/))
Jan 28 11:57:32 sugarcube python3[8553]: [I 2021-01-28 11:57:32.737 JupyterHub proxy:400] Adding default route for Hub: / => http://127.0.0.1:15001
Jan 28 11:57:32 sugarcube python3[8553]: [I 2021-01-28 11:57:32.736 JupyterHub proxy:320] Checking routes
Jan 28 11:57:32 sugarcube python3[8553]: [I 2021-01-28 11:57:32.732 JupyterHub service:121] Spawning /opt/tljh/hub/bin/python3 -m jupyterhub_idle_culler --timeout=600 --cull-every=60 --concurrency=5 --max-age=0
Jan 28 11:57:32 sugarcube python3[8553]: [I 2021-01-28 11:57:32.729 JupyterHub service:339] Starting service 'cull-idle': ['/opt/tljh/hub/bin/python3', '-m', 'jupyterhub_idle_culler', '--timeout=600', '--cull-every=60', '--concurrency=5', '--max-age=0']
Jan 28 11:57:32 sugarcube python3[8553]: [I 2021-01-28 11:57:32.729 JupyterHub app:2662] Starting managed service cull-idle
Jan 28 11:57:32 sugarcube python3[8553]: [I 2021-01-28 11:57:32.729 JupyterHub app:2647] Hub API listening on http://127.0.0.1:15001/hub/
Jan 28 11:57:32 sugarcube python3[8553]: [I 2021-01-28 11:57:32.728 JupyterHub app:2611] Not starting proxy
Jan 28 11:57:32 sugarcube python3[8553]: [I 2021-01-28 11:57:32.725 JupyterHub app:2402] Initialized 1 spawners in 0.024 seconds
Jan 28 11:57:32 sugarcube python3[8553]: [I 2021-01-28 11:57:32.723 JupyterHub app:2069] **** still running
Jan 28 11:57:32 sugarcube python3[8553]: [I 2021-01-28 11:57:32.667 JupyterHub app:1718] Not using allowed_users. Any authenticated user will be allowed.
Jan 28 11:57:32 sugarcube python3[8553]: [W 2021-01-28 11:57:32.622 JupyterHub configurable:168] Config option `blacklist` not recognized by `NativeAuthenticator`.
Jan 28 11:57:32 sugarcube python3[8553]: [I 2021-01-28 11:57:32.602 JupyterHub app:1412] Loading cookie_secret from /opt/tljh/state/jupyterhub_cookie_secret
Jan 28 11:57:32 sugarcube python3[8553]: [I 2021-01-28 11:57:32.595 JupyterHub app:2363] Using Proxy: jupyterhub_traefik_proxy.toml.TraefikTomlProxy-0+unknown
Jan 28 11:57:32 sugarcube python3[8553]: [I 2021-01-28 11:57:32.595 JupyterHub app:2363] Using Spawner: tljh.user_creating_spawner.UserCreatingSpawner
Jan 28 11:57:32 sugarcube python3[8553]: [I 2021-01-28 11:57:32.595 JupyterHub app:2363] Using Authenticator: nativeauthenticator.nativeauthenticator.NativeAuthenticator
Jan 28 11:57:32 sugarcube python3[8553]: [I 2021-01-28 11:57:32.594 JupyterHub app:2332] Running JupyterHub version 1.2.2
Jan 28 11:57:31 sugarcube systemd[1]: Started jupyterhub.service.
Jan 28 11:57:31 sugarcube systemd[1]: Stopped jupyterhub.service.
Jan 28 11:57:31 sugarcube python3[8406]: [I 2021-01-28 11:57:31.792 JupyterHub app:2483] ...done
Jan 28 11:57:31 sugarcube python3[8406]: [I 2021-01-28 11:57:31.791 JupyterHub app:2466] I didn't start the proxy, I can't clean it up
Jan 28 11:57:31 sugarcube python3[8406]: [I 2021-01-28 11:57:31.791 JupyterHub app:2458] Leaving single-user servers running
Jan 28 11:57:31 sugarcube python3[8406]: [I 2021-01-28 11:57:31.788 JupyterHub app:2446] Cleaning up 1 services...
Jan 28 11:57:31 sugarcube python3[8406]: [C 2021-01-28 11:57:31.786 JupyterHub app:2810] Received signal SIGTERM, initiating shutdown...
Jan 28 11:57:31 sugarcube systemd[1]: Stopping jupyterhub.service...
Jan 28 11:57:19 sugarcube python3[8406]: [I 2021-01-28 11:57:19.000 JupyterHub log:181] 200 POST /hub/api/users/test/activity (test@127.0.0.1) 32.16ms
Jan 28 11:57:01 sugarcube python3[8406]: [I 2021-01-28 11:57:01.424 JupyterHub log:181] 200 GET /hub/api/users (cull-idle@127.0.0.1) 25.71ms
Jan 28 11:56:01 sugarcube python3[8406]: [I 2021-01-28 11:56:01.425 JupyterHub log:181] 200 GET /hub/api/users (cull-idle@127.0.0.1) 26.78ms
Jan 28 11:55:13 sugarcube python3[8406]: [W 2021-01-28 11:55:13.873 JupyterHub log:181] 404 GET /hub/jupyter (test@***.***.***.***) 66.21ms
Jan 28 11:55:13 sugarcube python3[8406]: [I 2021-01-28 11:55:13.787 JupyterHub log:181] 302 GET /jupyter -> /hub/jupyter (@***.***.***.***) 1.62ms

[consideRatio]

I'll close this as a duplicate of #722, so it's not a zombie proxy configuration, it is more like a never updated proxy configuration! Thank you for providing such a thorough report, it really helps with maintenance efforts!!!