Remove duplication of GraphLayout scheduler plugin

[douglasdavis]

Following up on a discovery in #5267: the GraphLayout plugin is duplicated in the scheduler plugins. The latest release uses uuid to generate a unique name for each instance to avoid the duplication warning; we should be able to remove this work around.

[jrbourbeau]

Thanks for tracking this @douglasdavis. Just checking in, do you have the bandwidth to look into this? No worries if not

[douglasdavis]

James Bourbeau ***@***.***> writes:

Thanks for tracking this @douglasdavis. Just checking in, do you have the bandwidth to look into this? No worries if not

Yes! Will be digging in today

[douglasdavis]

Still digging but jotting down what I've tracked down so far.

TaskGraph.__init__ creates a GraphLayout instance and adds it to the scheduler plugins.

In distributed/dashboard/scheduler.py, the applications dictionary has two places where a TaskGraph is created:

distributed/distributed/dashboard/scheduler.py

Line 53 in baca13b

"/graph": graph_doc,

distributed/distributed/dashboard/scheduler.py

Line 60 in baca13b

"/individual-graph": individual_doc(TaskGraph, 200),

graph_doc creates a TaskGraph instance and individual_doc creates an instance of the class passed as the first argument.

test_simple loops over that dictionary triggering the creation of two TaskGraph instances:

distributed/distributed/dashboard/tests/test_scheduler_bokeh.py

Lines 62 to 66 in baca13b

	for suffix in applications:
	response = await http_client.fetch(f"http://localhost:{port}{suffix}")
	body = response.body.decode()
	assert "bokeh" in body.lower()
	assert not re.search("href=./", body) # no absolute links

What you tried in #5267, to only create a GraphLayout plugin in TaskGraph.__init__ if scheduler.plugins didn't already contain a plugin with name graph-layout, is the first solution I thought of. But it looks like that wasn't a good fix. Will keep trying some alternatives.

[douglasdavis]

Since these are two different dashboard endpoints would it make sense to have two GraphLayout plugin instances? In that case we can just pass a kwarg to TaskGraph.__init__ to give its GraphLayout a name. @jrbourbeau what do you think?