Adding HTTP metrics collection to Kibana integration #6169

ymao1 · 2023-05-11T15:56:03Z

What does this PR do?

Updates the Kibana integration to use the metricbeat HTTP module to collect background task utilization metrics from the Kibana API endpoint added in this PR.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

[ ]

How to test this PR locally

Run the stack using elastic-package: elastic-package stack up --version=8.9.0-SNAPSHOT -v -d
Build the package:
- cd integrations/packages/kibana
- elastic-package build
Deploy the package: elastic-package stack up -v -d --services package-registry
Navigate to https://localhost:5601/app/integrations/browse
Select and add the Kibana integration
Verify you see the new HTTP metrics background task utilization integration
Configure the integration for host https://kibana:5601 and username/password elastic:changeme
Verify you see a datastream for metrics-kibana.background_task_utilization-default
Verify you see data in the datastream (GET metrics-kibana.background_task_utilization-default/_search) with the kibana.background_task_utilization.stats.value.load field populated

Related issues

Collect background task autoscaling metrics using Elastic Agent kibana#155779

Screenshots

elasticmachine · 2023-05-11T16:07:58Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-05-15T11:58:56.352+0000
Duration: 28 min 9 sec

Test stats 🧪

Test	Results
Failed	0
Passed	31
Skipped	0
Total	31

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

elasticmachine · 2023-05-11T17:40:12Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`2/2`)	💚
Files	100.0% (`4/4`)	💚
Classes	100.0% (`4/4`)	💚
Methods	95.238% (`40/42`)	👎 -0.916
Lines	87.234% (`82/94`)	👎 -12.766
Conditionals	100.0% (`0/0`)	💚

ruflin · 2023-05-12T08:57:40Z

packages/kibana/_dev/build/docs/README.md

@@ -8,7 +8,7 @@ If the Kibana instance is using a basepath in its URL, you must set the `basepat

 ## Compatibility

-The `kibana` package works with Kibana 8.5.0 and later.
+The `kibana` package works with Kibana 8.8.0 and later.


I assume this change is because of the new dataset? All the other dataset still work for the older versions?

If the compatibility version in the manifest has changed then you cannot install this (and later) versions of the package on that cluster.

Yes, this change is because the background task HTTP API endpoint is only available in Kibana 8.8.0+ but all other datasets should still work for older versions.

Actually realized that the Kibana PR was merged for 8.9 not 8.8 so updated again for 8.9

ruflin · 2023-05-12T08:59:31Z

packages/kibana/data_stream/background_task_utilization/sample_event.json

+            "version": "20.04.6 LTS (Focal Fossa)"
+        }
+    },
+    "http": {


I suggest to put this under kibana.background_task_utilization instead.

@jsoriano @joshdover As we have dataset now, I wonder if we still need this prefixing of metrics? Where should we be heading here?

ECS recommends actually using a prefix to avoid conflicts with future additions to the common schema: https://www.elastic.co/guide/en/ecs/current/ecs-custom-fields-in-ecs.html#_proper_names

Since this is a fairly specific metric that I don't see getting added to ECS, I think a prefix might make sense here.

I'm not sure I understand what the recommendation is @joshdover, but to chip in, for the other data streams we put them under {product}.{data_stream}.{metric} so I'd prefer to stick to that unless that would cause problems.

I think we all made the same recommendation only with different words?

I agree that kibana.background_task_utilization makes sense! Looking at the metricbeat HTTP docs, I only see an option for specifying namespace which seems to automatically add it under the http field (http.${namespace}. Would I need to specify an ingest pipeline to rename it?

You can either do it with a rename in the agent or in the ingest pipline. Ideally, you could just specific the prefix :-( Worth adding a beats issue to get this fixed long term?

I added the rename in the agent and opened an issue in beats: elastic/beats#35443

ruflin · 2023-05-12T09:00:10Z

packages/kibana/manifest.yml

+            show_user: true
+            default:
+              - http://localhost:5601
+          - name: username


Should we recommend usage of API keys?

Maybe...

But, the Metricbeat module that powers the other data streams here do not support using API keys (see elastic/beats#29271), which might lead to some confusion?
Ideally we would prioritize adding API key support for the Metricbeat modules and then add it into each Integration as well.

ruflin · 2023-05-12T09:02:51Z

@miltonhultgren Would be great to get your review on this one. I'm also curious to hear your thoughts on the grouping. The Http metrics is a bit odd but it is caused by the auto grouping by input.

miltonhultgren · 2023-05-12T09:31:16Z

I'll have a look but I wanted to clarify (as usual) that this is meant only for Agent based collection?
Or is there a PR to also add this to the Metricbeat module to make it available on ESS?
Is there a dashboard for it that we might want to attach to the Stack Monitoring UI?

@ymao1

@ruflin The grouping makes sense in the way that it's separate from the things that power the Stack Monitoring UI. You could add more data streams to the same type though I'm not sure if the label for that group should be the name of the "module" or "technical interface" (the Stack Monitoring stuff also hits the HTTP API for metrics). In other places I think we just call it "Metrics".

The main point is that we're free to name this new group anything we want.

miltonhultgren

Overall this LGTM!

I had some questions/concerns about the data stream naming and the field prefixes and some nitpicks on the copytext.

miltonhultgren · 2023-05-12T09:49:04Z

packages/kibana/changelog.yml

@@ -1,4 +1,9 @@
 # newer versions go on top
+- version: "2.3.5"
+  changes:
+    - description: Add HTTP metrics to collect background task utilization metric


NIT:

Suggested change

- description: Add HTTP metrics to collect background task utilization metric

- description: Add background task utilization metric

Updated in 051981f

miltonhultgren · 2023-05-12T10:08:35Z

packages/kibana/docs/README.md

+
+### Background task utilization
+
+Background task utilization data stream uses the `/api/task_manager/_background_task_utilization` API of Kibana, which is available starting in 8.8.


NIT:

Suggested change

Background task utilization data stream uses the `/api/task_manager/_background_task_utilization` API of Kibana, which is available starting in 8.8.

The Background task utilization data stream uses the `/api/task_manager/_background_task_utilization` API of Kibana, which is available starting in 8.8.

or

Suggested change

Background task utilization data stream uses the `/api/task_manager/_background_task_utilization` API of Kibana, which is available starting in 8.8.

This data stream uses the `/api/task_manager/_background_task_utilization` API of Kibana, which is available starting in 8.8.

Updated in 051981f

miltonhultgren · 2023-05-12T10:10:23Z

packages/kibana/docs/README.md

+| timestamp |  | alias |
+
+
+An example event for `background_task_utilization` looks as following:


NIT:

Suggested change

An example event for `background_task_utilization` looks as following:

An example event for `background_task_utilization` looks as follows:

This document is auto-generated using the event placeholder {{event "background_task_utilization"}} in packages/kibana/_dev/build/docs/README.md so I can't change this wording without changing the auto-generation, which will likely change all the docs files in this repo 🙈

miltonhultgren · 2023-05-12T10:14:05Z

packages/kibana/data_stream/background_task_utilization/manifest.yml

+type: metrics
+title: Kibana background task utilization metrics
+release: beta
+dataset: kibana.http_metrics.background_task_utilization


Suggested change

dataset: kibana.http_metrics.background_task_utilization

dataset: kibana.background_task_utilization

I don't think it's needed to add the type into the dataset name. We've added that to the Stack Monitoring legacy streams to not block the usage of that name for when/if new data streams are introduced for the "same" data but natively to the Integration.

Updated in 051981f

miltonhultgren · 2023-05-12T10:15:53Z

packages/kibana/data_stream/background_task_utilization/manifest.yml

+    title: Kibana background task utilization metrics
+    description: Collect Kibana background task utilization metrics


It would be nice if we could include some information about what those metrics are and what they are useful for.

NIT:
Maybe we don't need to write Kibana everywhere since the context should be clear to the user from being on the Kibana integration page?

Updated in 051981f

miltonhultgren · 2023-05-12T10:17:44Z

packages/kibana/data_stream/background_task_utilization/sample_event.json

+            "version": "20.04.6 LTS (Focal Fossa)"
+        }
+    },
+    "http": {


I'm not sure I understand what the recommendation is @joshdover, but to chip in, for the other data streams we put them under {product}.{data_stream}.{metric} so I'd prefer to stick to that unless that would cause problems.

ymao1 · 2023-05-12T11:57:20Z

I'll have a look but I wanted to clarify (as usual) that this is meant only for Agent based collection? Or is there a PR to also add this to the Metricbeat module to make it available on ESS? Is there a dashboard for it that we might want to attach to the Stack Monitoring UI?

@ymao1

@miltonhultgren Yes, this is only meant for agent-based collection. We did not update the Kibana module in metricbeat.

ymao1 · 2023-05-12T15:04:29Z

@miltonhultgren I had to update the fields mappings for some of the other datastreams for 8.9 to get the tests to pass: e8d1489

Is this the right way to do that?

miltonhultgren · 2023-05-15T06:47:29Z

@ymao1 Yes, the Integration system tests asserts that all fields in the ingested document have a mapping (which is good/bad) so that's the right way forward 👍🏼

…ger-utilization

elasticmachine · 2023-05-15T14:08:49Z

Package kibana - 2.3.5 containing this change is available at https://epr.elastic.co/search?package=kibana

Initial commit

d9df422

ymao1 added 4 commits May 11, 2023 13:07

Cleanup

1e0226e

Fixing checks

26a7a4f

Updating README:

4a984b1

Updating docs

f1675a5

ymao1 changed the title ~~Initial commit~~ Adding HTTP metrics collection to Kibana integration May 11, 2023

ymao1 mentioned this pull request May 11, 2023

Collect background task autoscaling metrics using Elastic Agent elastic/kibana#155779

Closed

Updating kibana version to 8.8

925903f

ruflin reviewed May 12, 2023

View reviewed changes

miltonhultgren approved these changes May 12, 2023

View reviewed changes

ymao1 added 2 commits May 12, 2023 08:49

PR feedback

051981f

Updating fields for 8.9

e8d1489

ymao1 added 2 commits May 15, 2023 07:53

Merge branch 'main' of github.com:elastic/integrations into task-mana…

c1ff363

…ger-utilization

Updating kibana version to 8.9

78df62d

ymao1 marked this pull request as ready for review May 15, 2023 11:59

ymao1 requested a review from a team as a code owner May 15, 2023 11:59

miltonhultgren approved these changes May 15, 2023

View reviewed changes

ymao1 merged commit 2935b0f into elastic:main May 15, 2023
3 checks passed

ymao1 deleted the task-manager-utilization branch May 15, 2023 13:47

ymao1 mentioned this pull request May 18, 2023

Investigate collecting background task load metric from Kibana using Metricbeat elastic/beats#35312

Closed

	- description: Add HTTP metrics to collect background task utilization metric
	- description: Add background task utilization metric


		### Background task utilization

		Background task utilization data stream uses the `/api/task_manager/_background_task_utilization` API of Kibana, which is available starting in 8.8.

	Background task utilization data stream uses the `/api/task_manager/_background_task_utilization` API of Kibana, which is available starting in 8.8.
	The Background task utilization data stream uses the `/api/task_manager/_background_task_utilization` API of Kibana, which is available starting in 8.8.

	Background task utilization data stream uses the `/api/task_manager/_background_task_utilization` API of Kibana, which is available starting in 8.8.
	This data stream uses the `/api/task_manager/_background_task_utilization` API of Kibana, which is available starting in 8.8.

		\| timestamp \| \| alias \|


		An example event for `background_task_utilization` looks as following:

	dataset: kibana.http_metrics.background_task_utilization
	dataset: kibana.background_task_utilization

		title: Kibana background task utilization metrics
		description: Collect Kibana background task utilization metrics

Adding HTTP metrics collection to Kibana integration #6169

Adding HTTP metrics collection to Kibana integration #6169

Conversation

ymao1 commented May 11, 2023 • edited

What does this PR do?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

elasticmachine commented May 11, 2023 • edited

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

elasticmachine commented May 11, 2023 • edited

🌐 Coverage report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ymao1 May 12, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ruflin commented May 12, 2023

miltonhultgren commented May 12, 2023 • edited

miltonhultgren left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ymao1 commented May 12, 2023

ymao1 commented May 12, 2023

miltonhultgren commented May 15, 2023

elasticmachine commented May 15, 2023

ymao1 commented May 11, 2023 •

edited

elasticmachine commented May 11, 2023 •

edited

elasticmachine commented May 11, 2023 •

edited

ymao1 May 12, 2023 •

edited

miltonhultgren commented May 12, 2023 •

edited