[APM] Gather limitations of Lens and requirements of APM

[kpatticha]

In 8.6 we introduced mobile and we used Lens for the most used widgets. Lens offers specific charts and supported limited options.

• Gather the limitations that affect APM of using Lens
• List APM requirements
  1. [APM] Mobile APM Most used widget #143501 (comment)

The goal is to initiate a discussion with Lens team to see if we can use Lens in the long run and if they can support our requirements.

Related to:
https://github.com/elastic/apm-dev/issues/823 (internal)
#143502
#143501
Figma Design explorations

[elasticmachine]

Pinging @elastic/apm-ui (Team:APM)

[kpatticha]

Lens provides the ability to run the same report against a different time window relative to the current one.
You can compare the two periods by adding a time shift. However, this has its own limitations.

For example, terms can’t be used along with time shifts because we can’t guarantee data will be returned from all shards.
In the document (internal) you can find more about the limitation and why it might cause inconsistencies.

Screen.Recording.2022-12-15.at.15.49.39.mov

Instead, Lens suggests using filters for consistent results

But for our use cases in mobile (most used X devices) won't be as useful as I was expected

[sorenlouv]

Thanks for the writeup @kpatticha ! I added a comment to the doc, although since Joe (author of doc) is no longer with the company I wonder if anyone is going to react to it.

[kpatticha]

Thanks, @sqren. I will ping the team about your comment.

[kpatticha]

we discussed it in UI weekly thus closing the ticket. We can revisit the possibility of using Lens in the future.

[kpatticha]

PROS

1. fast creation of charts
2. free options (inspect, explore in discover, open in Lens)
3. consistency in UI

CONS

1. Limited data processing and visualization options
2. Performance.
   • Lens charts take longer to render than our custom charts.
   • '>20' embeddable might impact performance
3. Comparison, time shift in Lens.
   • Terms and time-shits in the lens don't work well
   • Right now terms can’t be used along with time shifts because we can’t guarantee data will be returned from all shards
   • significant constraint as most of the charts we want to render are based on terms agg
4. today Lens embeddable is not powerful enough for what we want. (ex curated dashboards for apm customer) (Product)

[kpatticha]

I'm sharing more details here as requested

Limited data processing and visualization options

For example, the following component is not available in Lens

Another concern from the engineering side is that some of the features we support in APM are not available in Lens. e.x Progressive loading

Lens charts take longer to render than our custom charts.

Screen.Recording.2023-02-24.at.13.22.48.mov

That can be connected to the data view though for the given example we pass apm_static_index_pattern_id

Screen.Recording.2023-02-24.at.13.25.26.mov

[kpatticha]

Updates after the call @achyutjhunjhunwala and I had with Lens team

• control over how we fetch data with lens: it's requested from other teams and is on their road map but not with high priority atm.
• terms with time shift (comparison):  technical limitation, it's not that they don't want to fix or don't have the capacity to do so
• performance: they will take a look
• security is going to deliver charts with Lens embeddable in 8.8

[sorenlouv]

Thanks for the update @kpatticha !

terms with time shift (comparison): technical limitation, it's not that they don't want to fix or don't have the capacity to do so

Remind me: Is term-based time shifts something we need for version comparisons? If so, what about regular times shift (aka time based comparisons)? Are they available for us to use as a replacement for the existing comparison feature?

[gbamparop]

Thanks for the update, if there are open issues from the Lens team, could you please list them here?

[kpatticha]

Remind me: Is term-based time shifts something we need for version comparisons? If so, what about regular times shift (aka time based comparisons)? Are they available for us to use as a replacement for the existing comparison feature?

By "version comparison" do you mean service mention comparison? if so, I haven't looked closely at the details for the feature.

Regular times shift is available in Lens

[kpatticha]

@gbamparop

[Lens] Use Lens embeddable only for data fetching or only for rendering - #152567

[sorenlouv]

By "version comparison" do you mean service mention comparison? if so, I haven't looked closely at the details for the feature.

Regular times shift is available in Lens

@kpatticha Sorry if I was unclear. My question is: why do we need "terms with time shift (comparison)" as opposed to just time shifts?

[kpatticha]

Initially, we needed that for the mobile cases where we display the charts for the top 5 devices, app version, OS version.

However, we don't support comparison for these charts atm. 🤔

[sorenlouv]

Initially, we needed that for the mobile cases where we display the charts for the top 5 devices, app version, OS version.

That makes sense. I was wondering if there is anything blocking us from migrating the latency/throughput/failure rate charts to Lens.