Group Progress Dashboard y-ranges

[mrocklin]

I'm running a computation on 400 threads. I think that they're all pretty well saturated. The Task Stream plot stopped working well for me, so I thought I'd take @ian-r-rose 's Group Progress graph for a spin. It looks great!

But! The question I want to answer here is "am I saturating all of my cores". I have 400 cores and when I hover over the filled graph I see that I'm using almost 400 cores. Woot. But why is the y-range so much taller? I'd expect that using 400/400 cores (or n / n cores) to look like it was full of color with no whitespace. Also, it'd be good to have a y-axis I think.

@ian-r-rose thoughts?

[ian-r-rose]

Good question! The maximum y should be equal to the maximum number of threads seen during the duration of the timeseries:

distributed/distributed/dashboard/components/scheduler.py

Lines 3023 to 3026 in 2b23840

	# Possibly update the y range if the number of threads has increased.
	max_nthreads = max(self.plugin.nthreads)
	if self.root.y_range.end != max_nthreads:
	self.root.y_range.end = max_nthreads

Might there have been any scaling up/down that could have made it an unexpected value? Do you have a reproducer that I can poke at?

Also, it'd be good to have a y-axis I think.

Agreed.

[mrocklin]

Sometimes it's fine though. I suspect that maybe we don't currently fix the y-ranges but allow them to float to the largest historical value? (note, I'm now seeing Ian's response, which came in concurrently)

I propose that maybe we fix the y-range to the current number of threads. Thoughts?

[mrocklin]

This would look bad in a scale-down situation, but maybe that's rare enough that we don't care.

I'll put my current work up in a gist. It's fun, you might like it. Certainly not minimal though.

[ian-r-rose]

I propose that maybe we fix the y-range to the current number of threads. Thoughts?

I'd be fine with this. As you say, it might look kind of wonky if there is a lot of scaling, but that's probably a case we don't need to optimize for right now.

[mrocklin]

I think that people probably scale up more often than they scale down.

[mrocklin]

I misunderstood the original plot. I thought that the y-range was actually at 1000 or so. It turns out that the y-range was actually pretty close to where it should have been.

I'd be inclined to restrict this issue to just "add y-axis". I think that that would have resolved my issue.

[mrocklin]

I'm also having a great time with this plot by the way. It's really helped me to understand what's going on. The separation between the groups within the same prefix is actually giving me more information than the task stream plot. (I'm looking for vertical separation between groups). This has been great. Thank you.