"Monitor" tab for health metrics

[albertteoh]

Requirement - what kind of business use case are you trying to solve?

The main proposal is documented in: jaegertracing/jaeger#2736.

The motivation is to help identify interesting traces (high qps, slow or erroneous) without knowing the service or operations up-front.

Use cases include:

• Post deployment sanity checks across the org, or on known dependent services in the request chain.
• Monitoring and root-causing when alerted of an issue.
• Better onboarding experience for new users of Jaeger UI.
• Long-term trend analysis of QPS, errors and latencies.
• Capacity planning.

Proposal - what do you suggest to solve the problem or improve the existing situation?

Add a new "Monitoring" tab situated after "Compare" containing service-level request rates, error rates, latency and impact (= latency * request rate to avoid "false positives" from low QPS endpoints with high latencies).

The data will be sourced from jaeger-query's new metrics endpoints.

As the jaeger-query metrics endpoints require opt-in to be enabled, the Monitor tab will have a sensible empty state, perhaps a link to documentation on how to enable metrics querying capabilities.

Workflow

The screen will open to a per-service level set of metrics sorted, by default, on Impact. Columns are configurable by the user with other latency percentiles available, among others. A search box will be available to filter on service names.

The user need only supply the time period to fetch metrics on (similar to Find Traces), defaulting to a 1 hour lookback.

Note the user is not required to define the step size (the period between data points), at least in this iteration, to keep the user experience as simple as possible. Instead we propose to define the step size based on a sensible heuristic based on the query period and/or the width of the chart. For example:

• < 30m search period -> 15s step
• < 1h search period -> 1m step, etc.

There are two possible actions from here in this tab:

• Click on a service to drill down to per-operation metrics.
• Click on "View all traces" to go to the Search tab with the service pre-populated and Operation filter set to "all".

Service metrics page

If drilling down into the service-level metrics, the page will show a summary of the RED metrics at the top along with the per-operation equivalent metrics as with the per-service metrics above. Also similarly, there will be a search box to filter on operations, and the user has the option to "View all traces" for a given operation.

Search tab

The search tab will be the final stage in the workflow (except of course if going back to a previous state), which is pre-populated with the service and/or operation as well as the search period.

The search period will be sticky between each of these screens to maintain consistency in search results.

Demo

Courtesy of @Danafrid.

Screen.Recording.2021-04-14.at.11.52.58.mov

Any open questions to address

• Any suggestions on charting libraries use for the larger detailed charts and the smaller row-level graphs in the table views?
• Any requirement to maintain consistency with the trace statistics table view?
• What is the preferred behaviour when a large number of services/operations are returned?
  • Show the top n results ordered by Impact by default? What if the user sorts on a different metric like errors? Just sort on the current n results or refetch from jaeger-query?
  • Show everything?
  • Paginate? (probably want to avoid this as it would require maintaining state in UI or jaeger-query)

[yurishkuro]

@albertteoh maybe we should move this to the main repo since it spans frontend and backend work.

[yurishkuro]

or at least we should have a master ticket in the main repo

[albertteoh]

Yup sure, I'll move this to the main repo as suggested, and will make this the master ticket as well, since this issue will drive the requirements for UI and backend work.

[albertteoh]

Closing in favour of master ticket: jaegertracing/jaeger#2954