[Telemetry] Track event loop delays on the server

[Bamieh]

Summary

Reports an array of daily histograms of delays in the event loop.

Uses perf_hooks.monitorEventLoopDelay to track the histogram with the default resolution of 10ms.

Event loop delays data is aggregated daily in a histogram per process ID.

The histogram is stored and updated in a saved object every 1 hour. This means that we'd maximum lose an hour worth of tracking on server shutdown.

Example payload

"event_loop_delays": {
  "daily": [
    {
      "min": 9093120,
      "max": 53247999,
      "mean": 11993238.600747818,
      "exceeds": 0,
      "stddev": 1168191.9357543814,
      "percentiles": {
        "50": 12607487,
        "75": 12615679,
        "95": 12648447,
        "99": 12713983
      },
      "processId": 55097
    },
    {
      "min": 9223372036854776000,
      "max": 0,
      "mean": null,
      "exceeds": 0,
      "stddev": null,
      "percentiles": {
        "50": 0,
        "75": 0,
        "95": 0,
        "99": 0
      },
      "processId": 54099
    },
  ]
}

Schema

export interface EventLoopDelaysUsageReport {
  daily: Array<{
    processId: number;
    min: number;
    max: number;
    mean: number;
    exceeds: number;
    stddev: number;
    percentiles: {
      '50': number;
      '75': number;
      '95': number;
      '99': number;
    };
  }>;
}

Notes

Once we have node v15.9 we can utilize the native Histogram provided by node to bake histograms into usage_counters to provide a histogram api in addition to the totals counter.

Context: #101283

[mshustov]

thus, we start collecting data 1 min after the start lifecycle of kibana_usage_collection plugin.

[Bamieh]

The monitoring of the event loop starts immediately. The first collection of the histogram happens after 1 minute. The daily histogram data is updated every 1 hour.

[mshustov]

The same comment about moment usage when the standard library should do the trick:

const date = new Date();
console.log(`${date.getDate()}_${date.getMonth()}_${date.getFullYear()}`);

[Bamieh]

I prefer using moment for dates especially when it comes to formatting etc. .format('DDMMYYYY') is a lot more readable and less error prone than the suggestion above IMO.

[mshustov]

is a lot more readable

I do agree it's more readable, I am just not sure that using moment for such simple functionality justifies the performance penalty. Yeah, it's not significant per se, but it's a road to death by a thousand cuts.

[Bamieh]

I'm generally against focusing on micro optimizations. There are a lot of bigger performance bottlenecks in an i/o project like kibana.

I dont think kibana would ever suffer in performance because JS can execute a moment function 200k times per second while a native date would execute 800k times. I'd always favor readablity and maintanablity in these cases. Especially since we have moment integrated in every place where we use dates even in eui components.

I did however replace moment.now() with Date.now() following your advice

[pgayvallet]

NIT: technically, being inside a subscription, I think this await does nothing. Not sure it would even protect against unresolved rejections?

[Bamieh]

Technically it is exactly the same as without await in this case since it is inside a .subscribe(..). I've just added it to explicitly denote that a Promise is expected here, also if we want to add some logic in the future under this function it might be easy to miss adding await

[mshustov]

optional nit: it's easy to forget updating the comment, so self-docuemented code FTW:

const Min = 60 * 1000;
const Hour = 60 * Min;
/** * Start rolling indices */
export const ROLL_DAILY_INDICES_INTERVAL = 24 * Hour
export const ROLL_INDICES_START = 5 * Min;

[lukeelmers]

Was trying to think if there was any way this could break with the plan for clustering, but AFAICT we should be fine as long as the pid is always reported with the metrics.

[kibanamachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: 0cb3650
• Storybooks Preview

Metrics [docs]

Saved Objects .kibana field count

Every field in each saved object type adds overhead to Elasticsearch. Kibana needs to keep the total field count below Elasticsearch's default limit of 1000 fields. Only specify field mappings for the fields you wish to search on or query. See https://www.elastic.co/guide/en/kibana/master/development-plugin-saved-objects.html#_mappings

id	before	after	diff
event_loop_delays_daily	-	2	+2

History

• 💔 Build #132765 failed 8dcd50e
• 💚 Build #132723 succeeded dc5f98b
• 💚 Build #131988 succeeded 0ad3bdb
• 💔 Build #131978 failed 7ef5929
• 💔 Build #130525 failed 73a92ad

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

[kibanamachine]

💚 Backport successful

Status	Branch	Result
✅	7.x

This backport PR will be merged automatically after passing CI.