Large amounts of notebook servers 50+ causes jupyter webapp slowness.

[boarder7395]

The way the jupyter web app is loaded causes slowness when the there are a lot of notebooks in a single namespace.

The notebook_dict_from_k8s_obj function calls the kubernetes api for events on each notebook in the namespace. With 50 notebooks we're seeing around 1.8-2 second response time. This is really bogging down the UI.

Relevant Code:

kubeflow/components/crud-web-apps/jupyter/backend/apps/common/routes/get.py

Lines 52 to 57 in d1c40f9

	@bp.route("/api/namespaces/<namespace>/notebooks")
	def get_notebooks(namespace):
	notebooks = api.list_notebooks(namespace)["items"]
	contents = [utils.notebook_dict_from_k8s_obj(nb) for nb in notebooks]
	return api.success_response("notebooks", contents)

kubeflow/components/crud-web-apps/jupyter/backend/apps/common/utils.py

Lines 122 to 143 in d1c40f9

	def notebook_dict_from_k8s_obj(notebook):
	cntr = notebook["spec"]["template"]["spec"]["containers"][0]
	server_type = None
	if notebook["metadata"].get("annotations"):
	annotations = notebook["metadata"]["annotations"]
	server_type = annotations.get("notebooks.kubeflow.org/server-type")
	return {
	"name": notebook["metadata"]["name"],
	"namespace": notebook["metadata"]["namespace"],
	"serverType": server_type,
	"age": notebook["metadata"]["creationTimestamp"],
	"last_activity": get_notebook_last_activity(notebook),
	"image": cntr["image"],
	"shortImage": cntr["image"].split("/")[-1],
	"cpu": cntr["resources"]["requests"]["cpu"],
	"gpus": process_gpus(cntr),
	"memory": cntr["resources"]["requests"]["memory"],
	"volumes": [v["name"] for v in cntr["volumeMounts"]],
	"status": status.process_status(notebook),
	"metadata": notebook["metadata"]
	}

kubeflow/components/crud-web-apps/jupyter/backend/apps/common/status.py

Lines 46 to 49 in d1c40f9

	# Try to extract information about why the notebook is not starting
	# from the notebook's events (see find_error_event)
	notebook_events = get_notebook_events(notebook)
	status_event, reason_event = get_status_from_events(notebook_events)

kubeflow/components/crud-web-apps/jupyter/backend/apps/common/status.py

Lines 160 to 175 in d1c40f9

	def get_notebook_events(notebook):
	name = notebook["metadata"]["name"]
	namespace = notebook["metadata"]["namespace"]
	nb_creation_time = dt.datetime.strptime(
	notebook["metadata"]["creationTimestamp"], "%Y-%m-%dT%H:%M:%SZ"
	)
	nb_events = api.list_notebook_events(name, namespace).items
	# User can delete and then create a nb server with the same name
	# Make sure previous events are not taken into account
	nb_events = filter(
	lambda e: event_timestamp(e) >= nb_creation_time, nb_events,
	)
	return nb_events

[boarder7395]

Created PR #7260

[boarder7395]

After doing additional testing I realized the primary problem was the read from disk for every request for every notebook. I expect the serialized API requests will also cause slowness but a better fix for that is true pagination.

[boarder7395]

I took a look at the code and fixing that backend code to allow pagination should be pretty straightforward. I'd be happy to pick up that work.

Although I do not have experience with angular js for implementing the front end changes.