[Monitoring] Usage collection

[chrisronline]

Relates to #73141

This PR adds telemetry data for Stack Monitoring from Kibana.

Here is a sample of the data sent to telemetry, which will most likely need to be transformed (via https://github.com/elastic/infra/issues/23272):

{
  "hasMonitoringData": true,
  "clusters": [
    {
      "clusterUuid": "PChw572_S52efCun1M0eNg",
      "license": "platinum",
      "metricbeatUsed": true,
      "elasticsearch": {
        "count": 1,
        "enabled": true,
        "metricbeatUsed": true
      },
      "kibana": {
        "count": 1,
        "enabled": true,
        "metricbeatUsed": true
      },
      "logstash": {
        "count": 1,
        "enabled": true,
        "metricbeatUsed": true
      },
      "beats": {
        "count": 2,
        "enabled": true,
        "metricbeatUsed": true
      },
      "apm": {
        "count": 1,
        "enabled": true,
        "metricbeatUsed": false
      }
    }
  ]
}

We are also tracking page views and setup mode clicks, which are available under stack_stats.kibana.plugins.ui_metric.stack_monitoring.

cc @alexfrancoeur

[jasonrhodes]

Can these page titles include dynamic names? You may not want to track every single unique page title/path a user visits, but rather just a subset of normalized ones without dynamic/variable segments.

I also recommend throwing in a 2nd track pageview but with a delay: 15000 on the call, which shows how many clusters stayed on the page some "significant amount of time" and will allow your page views to be compared to other observability ones which are currently all doing that. (We can eventually normalize that so it happens with one call across the board consistently, but for now we are keeping it all very manual.)

[chrisronline]

@jasonrhodes Great idea. I was wondering why that was there for the examples I went through.

In terms of dynamic data, this page titles will not contain that. I went through all the unique instance/node pages and ensured the tracking name just said something like kibana_instance. With that said, we still have the beats/apm pages in legacy formats, which I hope to resolve in #76946 before updating here

[kibanamachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: b01ed4d

Build metrics

async chunks size

id	value	diff	baseline
monitoring	1.2MB	-4.4KB	1.2MB

page load bundle size

id	value	diff	baseline
monitoring	303.1KB	+518.0B	302.6KB

distributable file count

id	value	diff	baseline
default	45852	+6	45846

Saved Objects .kibana field count

id	value	diff	baseline
monitoring-telemetry	2	+2	-

History

• 💔 Build #76605 failed 11e1077
• 💔 Build #75213 failed b45d372
• 💔 Build #74954 failed 811cbe6
• 💚 Build #74465 succeeded 5189bd2
• 💔 Build #74457 failed 2e341c6

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

[elasticmachine]

Pinging @elastic/stack-monitoring (Team:Monitoring)

[afharo]

Telemetry changes LGTM!

I'm so excited about this new collector! Thank you @chrisronline.

@mindbat Mind the schema changes in x-pack/plugins/telemetry_collection_xpack/schema/xpack_plugins.json

[afharo]

@chrisronline FYI: we are one approval left to merge this improvement in the schema to be more explicit about the arrays: #78141

If it gets merged before this PR, you might need to update the schema. Otherwise, I will update mine 🙂

[igoristic]

@chrisronline Great job! Everything looks good 👍

I noticed that we're doing fetchLicenseType for every "monitored" cluster, but what about the license of the cluster that's doing the "monitoring"? By using something like plugins.licensing, since it might not be set to monitoring itself in a remote environment

Probably a different metric/feature, but I thought it would be interesting to see which license type is on the monitoring cluster itself

[chrisronline]

@igoristic That's a good point actually. We are only reporting data for the monitored clusters, which may or may not include the monitoring cluster itself. I think we can address that in a follow up if necessary. cc @ravikesarwani

[chrisronline]

Backport:

7.x: ee7f104