Replace chp with traefik-proxy

[GeorgianaElena]

• Add / update documentation
• Add tests

Hi!

I replaced configurable-http-proxy with a proxy that uses traefik and a toml file to store the routes (rules.toml), regarding jupyterhub/traefik-proxy#32.
The proxy is available here.
Also, I managed to find a way to wait for the hub to be fully operational after a restart and got rid of the sleeps in the tests (the 1 sec sleep wasn't enough for traefik proxy)

assert 0 == await (await asyncio.create_subprocess_exec(*TLJH_CONFIG_PATH, 'reload')).wait()
# FIXME: wait for reload to finish & hub to come up
# Should be part of tljh-config reload
await asyncio.sleep(1)

In order to make sure the hub is ready after the restart, I first waited for jupyterhub service to be active and then I waited for the string "JupyterHub is now running at" to be available in jupyterhub's logs(since last restart time). @minrk, @yuvipanda, I'm not sure I chose the right approach for this, so a feedback would be highly appreciated.

Also, I don't understand why the test test_hub.py/test_user_admin_remove is failing. I noticed this test was also failing before this PR.
In the logs I see this:

[SingleUserNotebookApp auth:866] oauth state 'eyJ1dWlkIjogImZhYTAxZmRhODc4YTQ2ZjFhMGE0ZDMzMjE5Njc3YWExIiwgIm5leHRfdXJsIjogIi91c2VyLzhjYjk0ZTBlMzZkOTVjZjYvdHJlZSJ9' != None`
[SingleUserNotebookApp web:1667] 403 GET /user/8cb94e0e36d95cf6/oauth_callback?code=9ec9fca1-0b50-4b23-9692-ad1e222866d0&state=eyJ1dWlkIjogImZhYTAxZmRhODc4YTQ2ZjFhMGE0ZDMzMjE5Njc3YWExIiwgIm5leHRfdXJsIjogIi91c2VyLzhjYjk0ZTBlMzZkOTVjZjYvdHJlZSJ9 (127.0.0.1): oauth state does not match. Try logging in again.

[minrk]

This is great progress! It's super exciting to see this coming together.

[minrk]

This is great! I've tested this locally and it seems to work very well. I'm super pleased with the traefik-proxy so far.

@yuvipanda what do you think about using this unconditionally in tljh? A more complicated option could be to preserve the CHP implementation and allow users to pick the implementation, in case there are issues.

The only downside to traefik-proxy that I'm aware of is the lack of activity tracking. I'm working on an alternate version of that that would work for all proxies in jupyterhub/jupyterhub#2346

[yuvipanda]

@GeorgianaElena this is awesome work! I'm very happy to have it here :)

@minrk I think it's ok to switch to it unconditionally. Here's what I think we should do before that:

• Make sure upgrades work fine between CHP and this
• Run http://github.com/yuvipanda/hubtraf against this for a while with users coming and going to make sure it works fine.

How does that sound?

[minrk]

Sounds good. I'll run a couple of those tests.

[minrk]

I did some review with minor things to change, but this seems to work great!

I was able to run an upgrade using:

export TLJH_BOOTSTRAP_PIP_SPEC=git+https://github.com/GeorgianaElena/the-littlest-jupyterhub@replace_chp_traefik
curl https://raw.githubusercontent.com/GeorgianaElena/the-littlest-jupyterhub/replace_chp_traefik/bootstrap/bootstrap.py   | sudo -E python3

which left the following tasks:

1. make sure that the old configurable-http-proxy.service is stopped and deleted, if it exists
2. add passlib and jupyterhub-traefik-proxy to install_requires in setup.py

I opened #277 as a side issue, since bootstrap doesn't work if https is enabled, but disabling https during the upgrade and then re-enabling after worked fine. I had to do /opt/tljh/bin/hub/pip install passlib to get past the missing dependency.

I ran hubtraf with 4 CPUs and 8GB of RAM on GCP. At 60 users, I started to see failures which prompted jupyterhub/traefik-proxy#41 which I think was the culprit (a race between the hub updating the routing table and traefik trying to read it). After installing that branch, 60 users ran without a single failure.

I'll look into parsing the results shortly.

[minrk]

@GeorgianaElena this is great! I think it's ready to go. I added a small change that fixed the issue, which was that the jupyterhub_config.py wasn't loading the generated password. I also moved the generation of the auth_basic bit to traefik.py where we generate that config file, so that secrets get loaded as part of the 'user' configuration instead of the 'default' configuration, which removed the need for mocking out the generate_traefik_api_credentials

[GeorgianaElena]

Thanks a lot for your help, @minrk!

[minrk]

Running hubtraf with jupyterhub-traefik-proxy 0.1.2 (needs jupyterhub/traefik-proxy#51) resulted in 0 failures for 20, 40, and 60 users, so I think we are set here.

[minrk]

Great job, @GeorgianaElena!

[yuvipanda]

This is awesome, @GeorgianaElena / @minrk! How would you feel about writing a blog post about this work?

[GeorgianaElena]

Thank you @minrk, @yuvipanda!
@yuvipanda , I would really like to give it a try.

[yuvipanda]

@GeorgianaElena awesome. If you can make an account on medium.com and tell us, we can give you access to send stories to blog.jupyter.org. Then you can write it, a bunch of us can help edit it and publish it. How does that sound?

/cc @choldgraf who is amazing and helps with these things.

[choldgraf]

definitely more than happy to help writing/editing/whatever's useful!

[minrk]

@GeorgianaElena's draft is submitted to the Jupyter blog if folks want to give it a review. I think we can post it early next week.