Meta issue: [APM] Service overview: Introduce time-series comparison

[formgeist]

Summary

Related design issue elastic/apm#300

The service overview introduces a new view for the selected service that encompasses a lot of the service data from existing views like metrics, transactions, and errors. The primary goal of the view is to aid in troubleshooting a service from a birds eye view rather than immediately diving into the Transactions, as that's the current landing page for a selected service. There's more information about the user stories and goals of the view in the design issue linked above.

Please note: I've marked this as a research ticket because there's still a few outstanding decisions to make on what range we're implementing as a comparison.

Time-series comparison

The general idea of the time-series comparison is to provide the user with more background on the currently selected time range. It's typically part of the analysis to investigate how the service was behaving a week ago or similar to give an indication of whether something has been slowly increasing or it's a sudden spike.

We want to make it simple for our users to select a time range and immediately have a comparison on the charts as a visual aid. This means adding an option next to the existing time range selection that allows the user to turn on/off a comparison time-series plot on all the relevant time-series charts.

Comparison options: Yesterday, a week ago, and previous period.

The UI will contain a dropdown with the options "Yesterday" and "A week ago" (wording might change).
This dropdown is only applicable when the selected time range is less than 24 hours. The dropdown will be set to readOnly when the time range is above 24 hours. See below for details:

Time range is between 0 - 24 hours

• Dropdown is enabled
• choosing "Yesterday" will compare the current time range with an identical time range yesterday
• choosing "A week ago" will compare the current time range with an identical time range last week

Time range is between 24 hours - 1 week

• Dropdown is set to readOnly
• "A week ago" will is selected (similar comparison as above)
  • Example: if selected time range is "Last 2 days" the comparison will be "9 days ago - 7 days ago".
  • Example: if selected time range is "Last 7 days" the comparison will "14 days ago - 7 days ago"

Time range is > 1 week

• Dropdown is set to readOnly
• The selected option will show the absolute date range i.e. "01/02 - 01/28" (respecting the Kibana date format set) to clarify the comparison date range. The reason is that relative is a lot harder to understand once you go beyond "a week ago".
• Selected time range will be compared to identical time range immediate before.
  • Example: if selected time range is "January 10th - 18th" (8 days) the comparison will be "January 2nd - 10th".
  • Example: if selected time range is "Last 30 days" the comparison will be "60 days ago - 30 days ago".
  • Example: if selected time range is "Last 2 weeks" the comparison will be "4 weeks ago - 2 weeks ago".

We should add the comparison range to the URL params of the URL so it's easy for users to share a specific time and comparison for team members.

Visualizing the comparison range

The currently selected time range will be visualized the palette color matching the metric. The comparison period will feature as an area chart behind it with a grey fill and stroke. The comparison is also a thinner stroke than the current range.

[elasticmachine]

Pinging @elastic/apm-ui (Team:apm)

[formgeist]

Updated the issue description to describe the final decision on the comparison intervals. We'll be making two options available; yesterday and a week ago. Read more in the description.

[wylieconlon]

As part of the time offsets we'll need to do basic date math to present the times as if they are overlapping: this is something that should be shared, not exclusive to APM. Can it be put in a shared place in Kibana code? We already have one implementation of this logic in Timelion, and we need it as part of an upcoming esaggs feature that is intended for use in Lens.

[sorenlouv]

Looks like a decision was made to not have comparisons for date ranges longer than 7 days. I'm not necessarily against it but interested to hear the reasoning behind this.

[formgeist]

@sqren The reasoning was primarily trying to scope the comparison feature to an essentials-only feature in order to get this in along-side the Service overview. We have a bit more time now, and I don't see any technical reasoning for not allowing 7+ day comparison other than we need to reconsider our option labels. "A week ago" loses its clarity when you've selected a range longer than 7 days. We can consider relabelling this to "previous period" but it's less defined.

[formgeist]

@sqren The reasoning was primarily trying to scope the comparison feature to an essentials-only feature in order to get this in along-side the Service overview. We have a bit more time now, and I don't see any technical reasoning for not allowing 7+ day comparison other than we need to reconsider our option labels. "A week ago" loses its clarity when you've selected a range longer than 7 days. We can consider relabelling this to "previous period" but it's less defined.

This jogged my memory a bit, I think I misunderstood the details of the description of the different scenarios. We should not disable comparison even beyond 7 days, but we will disable (set to readOnly) the option within the dropdown to display the absolute date range of the comparison. I've updated the description with examples to hopefully clarify this further.

cc @cauemarcondes @alex-fedotyev

[cauemarcondes]

Data fetching strategy

We're going to have one single API that returns the normal data and the comparison data. To reduce the complexity of the Pull Request, a few are going to be open that handles each part of what is described above.

We are going to refactor some apis in order to make the comparison available:

PR: #88946

• Remove the sparklines from the transaction table API (GET /api/apm/services/{serviceName}/transactions/groups/overview)
• Create a new API that returns the sparklines

PR:

• Remove the sparklines from the service inventory table API (GET /api/apm/services)
• Create a new API that returns the sparklines

APIs that must be changed adding comparison request:

• GET /api/apm/services/{serviceName}/transactions/charts/latency PR:
• GET /api/apm/services/{serviceName}/transactions/charts/throughput PR:
• GET /api/apm/services/{serviceName}/transactions/charts/error_rate PR:
• New API that returns transaction Sparklines PR:
• New API that returns Service Sparklines PR:

[cauemarcondes]

Linking all PRs that closes this meta issue:

• [APM] Break down transaction table api removing the sparklines #88946
• [APM] Break down error table api removing the sparklines #89138
• [APM] Adding comparison to throughput chart #90128
• [APM] Adding comparison latency chart #91339
• [APM] Adding comparison to transactions table #91435
• [APM] Telemetry: Add UI usage telemetry on the Timeseries comparison feature #91439
• [APM] Split All instances API adding a new API to return comparison statistics #94767
• [APM] add comparison to Instances latency distribution #97710
• [APM] Adding comparison data to the dependencies table #97843
• [APM] Service overview: Introduce time-series comparison #88665