Ambassador routes for new jupyter notebooks don't work

[jlewi]

Here's the service definition

apiVersion: v1
kind: Service
metadata:
  annotations:
    getambassador.io/config: |-
      ---
      apiVersion: ambassador/v0
      kind:  Mapping
      name: notebook_kf-jlewi_jlewi-1_mapping
      prefix: /notebook/kf-jlewi/jlewi-1
      rewrite: /kf-jlewi/jlewi-1
      timeout_ms: 300000
      service: jlewi-1.kf-jlewi:8888
      use_websocket: true

When I navigate to:

https://jlewi-0228.endpoints.cloud-ml-dev.cloud.goog/notebook/kf-jlewi/jlewi-1/

I get upstream connect error.

Is it supposed to be rewriting it to /kf-jlewi/jlewi-1

/cc @kimwnasptd @lluunn

[jlewi]

Port-forwarding seems to work.

kubectl -n kf-jlewi port-forward jlewi-1-0 8888:8888

[lluunn]

hmm probably should not rewrite to that.
One quick thing to try is to edit the service and change the rewrite (to "/"?)

[kimwnasptd]

Didn't have the time to properly inform about this one. If my understanding is correct there are three issues we need to tackle here. The service annotation and port 8888, the rewrite annotation should match the prefix and lastly we need to properly use the prefix inside the Jupyter Pod.

For the service annotation, we have to remove 8888. Initially the created Service for the Notebook was a Headless Service. But port mapping doesn't happen in Headless Services (traffic to Service's port 80 will not be mapped to the Endpoint's 8888). Also, Ambassador doesn't resolve the Service's Endpoints. It just sends the traffic to the Service and lets k8s handle the rest of the traffic's flow. From these two we couldn't just set the annotation to "service: " + instance.Name + "." + instance.Namespace because the traffic would be sent to the Headless Service's port 80 and end up in the Pod's port 80, instead of port 8888. This is why I asked @lluunn to include the port on the annotation. But then, in that PR, the Service became ClusterIP. So now the port mapping works just fine and we can just set the annotation to "service: " + instance.Name + "." + instance.Namespace, in the controller.

The rewrite annotation should match the prefix. Ambassador rewrites essentially put a prefix in front of the traffic handed to the Service. To determine what value we will use there, we need to discuss how the Jupyter Pod handles the prefix that it lives under.

By default Jupyter has all its resources under "/". If I GET "/" in Jupyter, it will redirect me to "/tree?". But this is not the behavior we want, because if I connect to http://CLUSTER/notebook/<ns>/<nb-name>/ then Jupyter will redirect me to http://CLUSTER/tree? rather than http://CLUSTER/notebook/<ns>/<nb-name/tree?. We need to make the Jupyter Pod aware of the prefix. This is solved with the NotebookApp.base_url
flag which was also used in the previous metacontroller implementation of the controller. By setting this flag to be /notebook/<ns>/<nb-name>/ (the prefix), then all the Jupyter's resources will live under that prefix and it will redirect us to the correct URLs.

But if we don't use the rewrite annotation, then http://CLUSTER/notebook/<ns>/<nb-name>/ would effectively be written to http://<nb-service>/. But because we used the base_url flag, the resources live under the prefix and not under "/", thus we get a 404. Finally to solve this we will need to set the rewrite annotation to match the prefix and base_url. This way the Jupyter Pod will see requests to /notebook/<ns>/<nb-name>/ which is where it is configured to have its resources from the base_url flag.

To sum up, we need to:

• Remove the port from the service annotation for Ambassador
• Set the rewrite annotation to match the prefix
• Set the base_url to be the same with the prefix and rewrite.

We still need to discuss how to properly set the base_url flag in the Jupyter Pod. We should probably open a new issue for it.