Dashboard docs on /status page

[ncclementi]

• Closes Create dashboard documentation page #8627
• Tests added / passed
• Passes pre-commit run --all-files

This is an initial pass at a docs entry that documents the /status page of the dashboard. Please take a look at the descriptions as I might have missed something or explained it incorrectly.

Pending:

• Resolve The Task Processing section confusion:

While going over the color code for the Task Processing plot in the dashboard I saw that they depend on whether the worker is saturated or idle (green or red).
https://github.com/dask/distributed/blob/713204b9757065cc1ce6df3574957a79a9f56ea6/distributed/dashboard/components/scheduler.py#L1502-L1508
My understanding was that when white, the worker is idle/with no tasks, In what scenario do we see red?

As I'm reading the SchedulerState class and the definitions of saturated and idle are a bit confusing, and this is making it a bit hard to explain what the colors green and red actually mean.
https://github.com/dask/distributed/blob/922d987261637152636087d93bf5ca333e8faefb/distributed/scheduler.py#L1924-L1927

EDIT:
This seems to be a better definition of what idle and saturated are, I'll edit the docs based on this description
check_idle_saturated method that defines what saturated and idle are here for completeness https://github.com/dask/distributed/blob/922d987261637152636087d93bf5ca333e8faefb/distributed/scheduler.py#L3333

• CPU Utilization tab: Needs a better explanation, ideas are welcome.
• ALT text in all the figures. To add once settled on the images
• Any suggestions on design and or explanations are welcome.

cc: @jsignell @jacobtomlinson

[jacobtomlinson]

This looks to be a great start, thanks @ncclementi!

Is this page intended to replace the current one? If so should we consider where docs for things like the progress bars and performance reports should go?

Also do we want to mention how the Dashboard integrates with Jupyter Lab?

[ncclementi]

@jacobtomlinson regarding:

Is this page intended to replace the current one? If so should we consider where docs for things like the progress bars and performance reports should go?

I'm not sure if that is the intention. I think that performance reports and progress bars can/should go on a separate page, but I don't know where would that live. I think for now I'd leave the duplicated content since the current diagnostics page lives in distributed, while this new page is in dask/dask.

Also do we want to mention how the Dashboard integrates with Jupyter Lab?

I think this is a good point. I can add a section on Dashboard + Jupyter Lab

[jacobtomlinson]

I think for now I'd leave the duplicated content since the current diagnostics page lives in distributed, while this new page is in dask/dask.

That page does live in dask/dask.

[jsignell]

I think ideally there would be one top level diagnostics item, and it can either include the information about the local diagnostics, or it can have a toc tree pointing to another page (like we do in the dataframe section for instance).

[bryanwweber]

Hi Naty! This looks really excellent, thanks for putting in so much effort towards this huge topic. I hope you don't mind that I left you some detailed comments at this fairly early stage. I tried to look at this page from my current perspective with respect to Dask/Distributed. That is, a fairly new user who's run the examples and tutorial, so I've seen the dashboard and I want some more information from it 😄

At a high level, I think it would be worthwhile to define who the audience for this document is (and apologies if that's spelled out in the issue and I missed it). If this is intended to be a reference for experienced Dask users, then I think my comments can largely be ignored. However, if the audience is meant to be new-ish users, then I hope this feedback will be useful.

Aside from that, I think there is a ton of useful information here and I can't wait to see it live!

Thanks!

[ncclementi]

Thank you all for the feedback I've included most of the comments and suggestions. I still have some things to take care of but this is looking much better.

@ian-r-rose and @bryanwweber I added the HTML squares with their colors as we talked. Thank you Ian for the code suggestion, I found that there is an HTML entity for squares, so I used that instead of the border and filling initially proposed. The only thing I'm struggling with is how can we add alt text to those squares.

[ncclementi]

Update:

Folks, I took a crack at the ALT text but I think it needs some work. If you have any suggestions please comment on them. @scharlottej13 You've written a lot of alt text on the sprint, would you mind commenting on this?

There is an extra concern I have regarding ALT text, all the squares with color, that we added for example this one, when using a voiceover they all read as "black square" instead of a blue square for example. I found a workaround that I took from SO see https://stackoverflow.com/a/68195136/16824375 but I'm not sure this is the best solution. If anyone knows a better idea, let me know.

I think content-wise this is ready, I'm still not sure about where to the Dask JupyterLab extension section, as well as what size to choose for the headings. @jsignell Do you have any ideas/preferences on the organization

[scharlottej13]

Folks, I took a crack at the ALT text but I think it needs some work. If you have any suggestions please comment on them. @scharlottej13 You've written a lot of alt text on the sprint, would you mind commenting on this?

Thanks for the ping! Happy to take a look today.

[scharlottej13]

This looks great @ncclementi! Feel free to take or leave my alt-text suggestions. Based on the W3C guidance, I tried to keep them short and tried to explain the purpose of the figure, rather than describe it in detail.

[scharlottej13]

Suggested change

	:alt: The Byte stored pane showing blue and grey horizontal bar of the cluster memory, the blue portion
	corresponds to 12.6 GiB of memory while the grey portion represents the 750 MiB spilled to disc. The Bytes
	stored per worker pane showing 16 workers each of them with a horizontal bar. The first one from the top is
	orange and grey, the following three are orange, and the remaining twelve are blue. The orange and blue bars
	also have different levels of transparency.
	:alt: Two bar charts on memory usage. The top chart of total cluster memory shows a cluster with mostly managed memory usage in blue, with very little spilled to disk in grey. The bottom chart displays memory usage per worker, with a separate bar for each of the 16 workers. The first worker stands out as having roughly two-thirds unmanaged memory close to spilling to disk in translucent orange, and the rest spilling to disk in grey. The next three are close to spilling to disk, in orange, and the rest are mostly using managed memory that is not close to spilling to disk, in blue.

[ncclementi]

This suggestion I think it's a bit confusing because we don't explain in this document what are managed and unmanaged memory. I don't have a better solution yet, but I'll think about it.

[scharlottej13]

Suggested change

	in their individual color with different transparency levels. The "double" and "inc" bars have a grey background, and
	as individual colors that correspond with the task stream. The "double" and "inc" bars have a grey background, and

[ncclementi]

I wonder if we should mention level of transparency or not.

[ncclementi]

Folks, I think this is ready to go. If I can get the last review that would be great. There are probably things that can be improved but at this point, I think having this on the docs is better than nothing.

@jacobtomlinson I just requested your review as you have green button powers and a lot of docs knowledge, but happy to ping someone else if you don't have the bandwidth.

[jacobtomlinson]

Thanks for working on this @ncclementi!

This looks awesome and it looks like this has been through a ton of review already. I don't want to derail things by picking through it at this stage. We can always iterate in follow up PRs. Let's get this in.