ui: add a metrics dashboard for changefeeds

[danhhz]

These metrics have proven useful for introspecting changefeed behavior,
so adding them as a dashboard in case any users hit something that needs
debugging. Also will be useful for users to see what's going on in the
cluster's changefeeds.

Release note (admin ui change): CHANGEFEED metrics are now exposed in
the UI.

[cockroach-teamcity]

This change is 

[danhhz]

This is all entirely out of my wheelhouse and I entirely cargo-culted it from other dashboards, so definitely do a careful review, please.

The graph names and labels could probably use some wordsmithing (cc @rolandcrosby maybe?). Also a scrollbar is showing up in the dropdown now. I tried to look into it, but couldn't get the dropdown to stay open while i chrome inspector'd it and figured it'd be easier to ask.

[vilterp]

nit: indent with 2 spaces (sadly we don't have an autoformatter configured)

[danhhz]

Done

[rolandcrosby]

It's great to have a first stab at this -- thanks Dan! A few notes:

• What are the meanings of the different timings? I'd suggest different wording but I'm not sure what I'm looking at. The wording on the other two graphs is fine.
• Meta/docs issue: we should make sure that important changefeed metrics are documented (as in what they mean) and listed in our guides to setting up monitoring/Prometheus.
• How does this behave when there are multiple changefeeds? Are all the numbers aggregated across all changefeeds?
• Is there an easy way that we could add a table listing the currently active changefeeds? I know that information should be available on the jobs page but this might also be a good place to surface it.

[danhhz]

@vilterp Any thoughts on what to do about this?

[vilterp]

Hm, I don't remember what we did last time. Any prop on the dropdown that just lets you make it taller without having to scroll?

[danhhz]

@rolandcrosby

What are the meanings of the different timings? I'd suggest different wording but I'm not sure what I'm looking at. The wording on the other two graphs is fine.

Gotcha, let's sync offline on the timings names. I agree they're not great, but I've thought about it and don't have anything better yet.

Meta/docs issue: we should make sure that important changefeed metrics are documented (as in what they mean) and listed in our guides to setting up monitoring/Prometheus.

Changefeeds and admin ui actually have different docs writers, so first step is figuring out who 😀

How does this behave when there are multiple changefeeds? Are all the numbers aggregated across all changefeeds?

All aggregated. It's too expensive to store this per-changefeed, unfortunately.

Is there an easy way that we could add a table listing the currently active changefeeds? I know that information should be available on the jobs page but this might also be a good place to surface it.

I was able to pattern match this page, but that would likely require some original work (and visual design). If it falls to me to do it, I don't think it'll make the release.

[rolandcrosby]

For 2.2 we don't need to add anything besides the timeseries graphs to this page. We can revisit in a future release. Aside from the naming changes we discussed offline this LGTM.

[danhhz]

@rolandcrosby updated based on our convo, feel pretty good about the structure. Happy to push a wordsmithing PR if any changes fall out of documenting it.

[danhhz]

Thanks for the reviews!

@vilterp and I dug into the scrollbar in the dashboard dropdown but didn't find anything yet. He gave me the go ahead to merge this as-is and get it in the monday alpha, but I'll file an issue to revisit. (And @piyush-singh, he wanted me to give you a heads up about this.)

bors r=vilterp,rolandcrosby

[craig]

Build succeeded

• GitHub CI (Cockroach)