APM data shows telemetry being called ~30 times a day

[lizozom]

We recently enabled APM on the overview-aws-eu-west-1 kibana (QA environment used to monitor all the other QA deployments).
While analyzing the data, I noticed something interesting and after talking to @Bamieh I wanted to open an issue to track this.

This cluster is running 7.16.0 (meaning #117955 is already merged into it) and it seems like it's calling the telemetry API way more often than one would expect.

The latency of each call is 10-60s and they happen once about every hour.
Some contain failures.

APM data from past 7 days
Discover telemetry call transactions
One of the errors found in the telemetry request

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[afharo]

The repetition could be caused by #87846. And the fix will ship in 8.0.0.

In the meanwhile, I'd recommend upgrading to 7.16.2 to make use of the caching mechanism so it will save many ES requests.

After looking at the APM data, I also identified 2 potential improvements so we can reduce the amount of time it takes to generate the telemetry report:

• Usage tracking increments should be non-blocking #123151
• [Maps] Usage collector is refreshing the Data Views on every run #123154

I also added them to our Meta issue for Telemetry performance improvements #119466

[lizozom]

@afharo is this issue now redundant? Should I close it?

[afharo]

@lizozom I was going to ask the same since we covered most of it. I'll go ahead and close it. Feel free to reopen if needed.