We should have a dedicated endpoint for telemetry

[jasontedor]

Today telemetry is collected from a variety of endpoints:

• cluster stats /_cluster/stats
• cluster info /
• X-Pack usage /_xpack/usage
• license info /_license

There are two problems with this approach:

• first, it's unclear to our users what information is used for telemetry
• secondly, it's unclear internally that these endpoints are for telemetry, which means from time to time we consider making changes to these endpoints that would either remove information that is useful from a telemetry perspective, or would add information that would not be useful from a telemetry perspective

If we had a single dedicated endpoint that exposed telemetry information, it would clear up some of the confusion around this, and simplify the implementation within Kibana.

There are open questions around how we expose telemetry from X-Pack in a single endpoint that would be defined in core, but @rjernst has some ideas around this that sound promising here.

Relates #52975

[elasticmachine]

Pinging @elastic/es-core-infra (:Core/Infra/Core)

[afharo]

Hi! I wanted to reach out to y'all to know your thoughts about this: we are about to merge this PR in Kibana: elastic/kibana#90273

It defines the full schema (aka contract) that we send to the Remote Telemetry service. Since it includes data that we pass through from the endpoints listed in the description, if ES returns new keys in the future, Kibana CI tests will fail, and we'll need to manually add the new fields to the schema definitions in the following files:

• OSS: src/plugins/telemetry/schema/oss_root.json
• X-Pack: x-pack/plugins/telemetry_collection_xpack/schema/xpack_root.json

Once the dedicated telemetry endpoint is created, we'll rework this to be more automated.

What do you think?

[dakrone]

I ran into an issue with the way our current telemetry works related to the /_cluster/stats endpoint. The endpoint can calculate some very expensive stats (mapping and analysis stats), which is fine for occasional telemetry retrieval, but the same endpoint is used within monitoring. In this particular case it was causing a non-negligible amount of load on the master node due to calculating the unneeded expensive stats.

Having a separate telemetry endpoint would allow these expensive-to-calculate parts that are not needed for monitoring to live in a dedicated place and avoid the load when performing monitoring.

[rjernst]

@afharo Core/Infra discussed the Kibana PR mentioned. While we understand the desire and need to have a set schema, we think the schema for Elasticsearch should live in Elasticsearch, not in Kibana. Cross repo changes like proposed are very difficult from many perspectives (temporary CI breakages, awareness of need for changes, unintentional refactorings, etc). Additionally, the APIs currently hit by Kibana for telemetry are strongly outside the scope of stable APIs; this is why this dedicated endpoint issue exists. We are happy to revisit priority of the dedicated endpoint, but do not feel the proposed PR should be merged.

[afharo]

@rjernst I understand the concerns for the cross-repo dependency. I totally agree with the risks. IMO, the fact that Kibana is reporting telemetry on behalf of Elasticsearch is already a cross-product dependency on its own.

I'd like to CC the telemetry stakeholders so we manage the expectations and prioritisations: @alexfrancoeur @thesmallestduck @elastic/infra-telemetry