Move Prometheus docs to dask/distributed

[crusaderky]

• Twin of Move Prometheus docs from dask/dask distributed#7405
• Partially closes Instrumentation of SpillBuffer distributed#7351

[mrocklin]

I'm against this. There was an intentional move of important docs from distributed.dask.org (which no one sees) to docs.dask.org (which gets lots of traffic). If we want people to read these docs then they should stay here. I appreciate that this causes some pain because it's in a different repository.

If folks want to move things to distributed.dask.org then I'd first like to see some work done in making sure that those docs have visibility. Right now they don't.

[mrocklin]

I apologize if I didn't make this sufficiently clear earlier. I apologize for creating possibly wasted work.

[hendrikmakait]

@mrocklin: For a better understanding, what do you mean by visibility here?

From what I can see, @crusaderky kept a link in the side-nav of docs.dask.org that redirects users to the distributed docs. This should create some level of visibility, i.e., people going through the side-nav will stumble across this and click on it. If we want to keep an actual page for some reason, I'd suggest moving the actual metric descriptions (starting from Scheduler metrics) over to distributed and linking from the Available metrics section. For the latter case, I would appreciate an explanation of the difference would be to the former in terms of visibility. Is this about SEO?

[mrocklin]

what do you mean by visibility here?

50x more people visit docs.dask.org . This is probably because that's what we link to from everywhere else. In practice the other foo.dask.org pages get very little traffic. In some cases this is ok, such as, say mpi.dask.org because it's pretty niche. However I think that setup documentation is important for pretty much all Dask users to read, so I want it to live in docs.dask.org. When we moved it from distributed.dask.org to docs.dask.org it received far more traffic.

More broadly there is a question of how much we want to fragment/federate our documentation. My sense is that fragmentation is better for developers (everyone writes in the repo that they know they best, updates on code accompany updates on docs) but worse for users (it's harder to navigate across the federated documentation, harder to search, etc..)

From what I can see, @crusaderky kept a link in the side-nav of docs.dask.org that redirects users to the distributed docs. This should create some level of visibility, i.e., people going through the side-nav will stumble across this and click on it

I'll admit that this is nice. However, I'd still like to keep everything in the same place if possible. After someone clicks on prometheus what will they click on next? My guess is that they're more likely to want to click on something in docs.dask.org

I would like to retire distributed.dask.org in the future if possible, and move everything over to docs.dask.org. I went through this process a while ago with the most important pieces (futures, setup, coordination primitives, understanding performance) so that what's left on distributed.dask.org is mostly some pretty niche reference material. Prometheus is important. I want it on docs.dask.org.

[crusaderky]

@mrocklin sorry I am not sure I understand.
From a end-user perspective, nothing has changed in terms of how you get to the prometheus doc page.
The only thing that has changed is that, once you're there, the navigation bar switches to distributed.

Prometheus is specifically an optional component of distributed; there's nothing about it in dask.array, dask.dataframe, etc.

Finally, Prometheus is something that a total beginner of dask is not going to look at. At all. What a beginner may love to do is to open the Coiled Grafana page - a very different thing. They're not going to start tampering with starting another service (or figure out the integration with an existing service) to collect metrics on their non-Coiled cluster. That's typically what the sysadmin/devops in the same company will do.

We can totally have a tinker about unifying everything so that distributed just becomes a sub-section of dask.org (e.g. the navbar doesn't change), but that's a different discussion altogether.

[mrocklin]

From a end-user perspective, nothing has changed in terms of how you get to the prometheus doc page.
The only thing that has changed is that, once you're there, the navigation bar switches to distributed.

I understand. I think that it would be good if the navbar stayed as it is in docs.dask.org .

Prometheus is specifically an optional component of distributed; there's nothing about it in dask.array, dask.dataframe, etc.

Thank you. I'm aware of this.

Finally, Prometheus is something that a total beginner of dask is not going to look at. At all

I'm aware of this too.

I would like us to move towards to retiring distributed.dask.org .

[mrocklin]

Maybe I don't understand. Is there an advantage to moving this to distributed.dask.org that I'm not aware of? What is the context/motivation here?

[crusaderky]

As discussed offline, I've reinstated the summary. Only the detailed API documentation has been moved to distributed.

[hendrikmakait]

LGTM, thanks @crusaderky!