Move metrics to setup and add cgroup metrics #76730

joshdover · 2020-09-03T23:35:37Z

Summary

Adds a collected_at field to OpsMetrics
Adds cgroup data to OpsMetrics (based on the legacy implementation: https://github.com/elastic/kibana/blob/71b9dedfc4eadd97db2b6b0dc038d0f640babf7b/src/legacy/server/status/lib/cgroup.js)
Moves ops metrics API to setup + add replay effect to ensure metrics are collected and ready by the time the HTTP server is available (to be used in next PR)
Renames two config keys:
- cpu.cgroup.path.override => ops.cGroupOverrides.cpuPath
- cpuacct.cgroup.path.override => ops.cGroupOverrides.cpuAcctPath

Release Note

These two config keys have been renamed and the old names will no longer work as of 8.0:

cpu.cgroup.path.override => ops.cGroupOverrides.cpuPath
cpuacct.cgroup.path.override => ops.cGroupOverrides.cpuAcctPath

Checklist

Delete any items that are not applicable to this PR.

Documentation was added for features that require explanation or tutorials
Unit or functional tests were updated or added to match the most common scenarios

For maintainers

This was checked for breaking API changes and was labeled appropriately

elasticmachine · 2020-09-03T23:35:39Z

Pinging @elastic/kibana-platform (Team:Platform)

…ements

TinaHeiligers

kibana_usage_collection changes LGTM

pgayvallet · 2020-09-09T05:47:56Z

src/core/server/metrics/types.ts

-export interface MetricsServiceStart {
+export interface MetricsServiceSetup {


What did require the contract to also be exposed during setup?

In order to use it in HTTP routes in Core, this was much simpler. It could also be added tot he requesthandlercontext, but I felt this would pollute that public API a bit since this is only needed by 2 endpoints right now.

src/core/server/metrics/metrics_service.mock.ts

pgayvallet · 2020-09-09T05:58:21Z

src/core/server/metrics/collectors/types.ts

+  cpuacct?: {
+    /** name of this process's cgroup */
+    control_group: string;
+    /** cpu time used by this process's cgroup */
+    usage_nanos: number;
+  };
+
+  /** cpu cgroup metrics, undefined when not running in a cgroup */
+  cpu?: {


I know we had format constraint when we initially implemented the metrics data structure, so I guess it's the same here, but still asking:

Can't we have a single cgroup entry here, that would look like

cgroup - name - cpu - ... - cpuacct - ...

We definitely could and I would prefer that. My thinking here was that doing it this way makes it easier not to break the API and is consistent with the rest of the metrics API in that way. We should reformat these to be snakeCase and re-shaped a bit for 8.0, but I think it makes sense to do that for all of metrics at once, which felt out of scope here.

src/core/server/metrics/collectors/os.ts

src/core/server/metrics/collectors/cgroup.ts

…ements

kibanamachine · 2020-09-09T17:41:46Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 15ff0e7

Build metrics

distributable file count

id	value	diff	baseline
default	45473	+1	45472
oss	27219	+1	27218
total		+2

History

💚 Build #73189 succeeded 8d930c4
💔 Build #72829 failed ce6dd2d
💔 Build #72827 failed 1e43465
💔 Build #72498 failed db910eb

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

# Conflicts: # src/core/server/config/deprecation/core_deprecations.ts

* master: (25 commits) [bugfix] Replace panel flyout opens 2 flyouts (elastic#76931) clean up test (elastic#76887) [Enterprise Search] Update shared API request handler (elastic#77112) [Maps] convert ESAggSource to TS (elastic#76999) Add plugin status API - take 2 (elastic#76732) Adds lens as a readable saved object for read-only dashboard users (elastic#77067) Skip checking for the reserved realm (elastic#76687) Reporting/diagnostics (elastic#74314) Reporting/Test: unskip non-screenshot tests (elastic#77088) Move metrics to setup and add cgroup metrics (elastic#76730) [Enterprise Search] Add Overview landing page/plugin (elastic#76734) First pass. Change TS type. Update OpenAPI (elastic#76434) [CI] Balance xpack ci groups a bit (elastic#77068) [Security_solution][Detections] Refactor signal ancestry to allow multiple parents (elastic#76531) [Maps] convert MetricsEditor to TS (elastic#76727) IndexMigrator: fix non blocking migration wrapper promise rejection (elastic#77018) [Enterprise Search] Update config data endpoint to v2 (elastic#76970) [ML] Add decision path charts to exploration results table (elastic#73561) Bump eventemitter3 from 4.0.0 to 4.0.7 (elastic#77016) [Ingest Pipelines] Add descriptions for ingest processors K-S (elastic#76981) ...

* master: (41 commits) [bugfix] Replace panel flyout opens 2 flyouts (elastic#76931) clean up test (elastic#76887) [Enterprise Search] Update shared API request handler (elastic#77112) [Maps] convert ESAggSource to TS (elastic#76999) Add plugin status API - take 2 (elastic#76732) Adds lens as a readable saved object for read-only dashboard users (elastic#77067) Skip checking for the reserved realm (elastic#76687) Reporting/diagnostics (elastic#74314) Reporting/Test: unskip non-screenshot tests (elastic#77088) Move metrics to setup and add cgroup metrics (elastic#76730) [Enterprise Search] Add Overview landing page/plugin (elastic#76734) First pass. Change TS type. Update OpenAPI (elastic#76434) [CI] Balance xpack ci groups a bit (elastic#77068) [Security_solution][Detections] Refactor signal ancestry to allow multiple parents (elastic#76531) [Maps] convert MetricsEditor to TS (elastic#76727) IndexMigrator: fix non blocking migration wrapper promise rejection (elastic#77018) [Enterprise Search] Update config data endpoint to v2 (elastic#76970) [ML] Add decision path charts to exploration results table (elastic#73561) Bump eventemitter3 from 4.0.0 to 4.0.7 (elastic#77016) [Ingest Pipelines] Add descriptions for ingest processors K-S (elastic#76981) ...

…rok/new-patterns-component-use-array * 'master' of github.com:elastic/kibana: (39 commits) [APM] Always load esarchives from common (elastic#77139) [Ingest Manager] Handle Legacy ES client errors (elastic#76865) [Docs] URL Drilldown (elastic#76529) [bugfix] Replace panel flyout opens 2 flyouts (elastic#76931) clean up test (elastic#76887) [Enterprise Search] Update shared API request handler (elastic#77112) [Maps] convert ESAggSource to TS (elastic#76999) Add plugin status API - take 2 (elastic#76732) Adds lens as a readable saved object for read-only dashboard users (elastic#77067) Skip checking for the reserved realm (elastic#76687) Reporting/diagnostics (elastic#74314) Reporting/Test: unskip non-screenshot tests (elastic#77088) Move metrics to setup and add cgroup metrics (elastic#76730) [Enterprise Search] Add Overview landing page/plugin (elastic#76734) First pass. Change TS type. Update OpenAPI (elastic#76434) [CI] Balance xpack ci groups a bit (elastic#77068) [Security_solution][Detections] Refactor signal ancestry to allow multiple parents (elastic#76531) [Maps] convert MetricsEditor to TS (elastic#76727) IndexMigrator: fix non blocking migration wrapper promise rejection (elastic#77018) [Enterprise Search] Update config data endpoint to v2 (elastic#76970) ... # Conflicts: # src/plugins/es_ui_shared/static/forms/hook_form_lib/components/use_array.ts

* master: (38 commits) Reporting/Test: unskip non-screenshot tests (elastic#77088) Move metrics to setup and add cgroup metrics (elastic#76730) [Enterprise Search] Add Overview landing page/plugin (elastic#76734) First pass. Change TS type. Update OpenAPI (elastic#76434) [CI] Balance xpack ci groups a bit (elastic#77068) [Security_solution][Detections] Refactor signal ancestry to allow multiple parents (elastic#76531) [Maps] convert MetricsEditor to TS (elastic#76727) IndexMigrator: fix non blocking migration wrapper promise rejection (elastic#77018) [Enterprise Search] Update config data endpoint to v2 (elastic#76970) [ML] Add decision path charts to exploration results table (elastic#73561) Bump eventemitter3 from 4.0.0 to 4.0.7 (elastic#77016) [Ingest Pipelines] Add descriptions for ingest processors K-S (elastic#76981) [Metrics UI] Replace Snapshot API with Metrics API (elastic#76253) legacy utils cleanup (elastic#76608) [ML] Account for "properties" layer in find_file_structure mappings (elastic#77035) fixed typo Upgrade to Kea 2.2 (elastic#77047) a11y tests on spaces home page including feature control (elastic#76515) [ML] Transforms list: persist pagination through refresh interval (elastic#76786) [ML] Replace all use of date_histogram interval with fixed_interval (elastic#76876) ...

Move metrics to setup and add cgroup metrics

db910eb

joshdover added release_note:deprecation Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc v8.0.0 v7.10.0 labels Sep 3, 2020

joshdover requested a review from a team as a code owner September 3, 2020 23:35

joshdover added 3 commits September 6, 2020 11:57

Fix tests on linux + types

36eb999

Merge remote-tracking branch 'upstream/master' into kp/metrics-improv…

1e43465

…ements

Fix test types

ce6dd2d

joshdover requested a review from a team as a code owner September 6, 2020 18:36

joshdover added 2 commits September 8, 2020 12:02

Fix usage collection test

e544c4c

Merge remote-tracking branch 'upstream/master' into kp/metrics-improv…

8d930c4

…ements

TinaHeiligers approved these changes Sep 8, 2020

View reviewed changes

pgayvallet approved these changes Sep 9, 2020

View reviewed changes

afharo mentioned this pull request Sep 9, 2020

[Metricbeat][Kibana][status] metrics is optional elastic/beats#20956

Closed

6 tasks

joshdover added 2 commits September 9, 2020 09:15

Merge remote-tracking branch 'upstream/master' into kp/metrics-improv…

464162e

…ements

PR comments

15ff0e7

joshdover merged commit 9bc603e into elastic:master Sep 9, 2020

joshdover deleted the kp/metrics-improvements branch September 9, 2020 20:01

joshdover mentioned this pull request Sep 9, 2020

[7.x] Move metrics to setup and add cgroup metrics (#76730) #77108

Merged

joshdover added a commit to joshdover/kibana that referenced this pull request Sep 9, 2020

Move metrics to setup and add cgroup metrics (elastic#76730)

6f86c5f

# Conflicts: # src/core/server/config/deprecation/core_deprecations.ts

joshdover added a commit that referenced this pull request Sep 9, 2020

[7.x] Move metrics to setup and add cgroup metrics (#76730) (#77108)

f5b6980

pgayvallet mentioned this pull request Sep 10, 2020

add cGroupOverrides to the legacy config #77180

Merged

igoristic mentioned this pull request Sep 15, 2020

[Parity] Monitoring to include cpuacct and cpu in kibana_stats elastic/beats#21092

Closed

joshdover mentioned this pull request Sep 16, 2020

Make sure getOpsMetrics$ only replays lastest value #77673

Merged

1 task

weltenwort mentioned this pull request Sep 22, 2020

Kibana refuses to start if metrics collector can't access sysfs #78191

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Move metrics to setup and add cgroup metrics #76730

Move metrics to setup and add cgroup metrics #76730

joshdover commented Sep 3, 2020

elasticmachine commented Sep 3, 2020

TinaHeiligers left a comment

pgayvallet Sep 9, 2020

joshdover Sep 9, 2020

pgayvallet Sep 9, 2020

joshdover Sep 9, 2020

kibanamachine commented Sep 9, 2020

		export interface MetricsServiceStart {
		export interface MetricsServiceSetup {

Move metrics to setup and add cgroup metrics #76730

Move metrics to setup and add cgroup metrics #76730

Conversation

joshdover commented Sep 3, 2020

Summary

Release Note

Checklist

For maintainers

elasticmachine commented Sep 3, 2020

TinaHeiligers left a comment

Choose a reason for hiding this comment

pgayvallet Sep 9, 2020

Choose a reason for hiding this comment

joshdover Sep 9, 2020

Choose a reason for hiding this comment

pgayvallet Sep 9, 2020

Choose a reason for hiding this comment

joshdover Sep 9, 2020

Choose a reason for hiding this comment

kibanamachine commented Sep 9, 2020

💚 Build Succeeded

Build metrics

distributable file count

History