[MongoDB Atlas] Add Hardware data stream #9689

niraj-elastic · 2024-04-23T17:19:31Z

What does this PR do?

Added 1 data stream (Hardware Metrics).
Added data collection logic for the data streams.
Added the ingest pipeline for the data streams.
Mapped fields according to the ECS schema and added Fields metadata in the appropriate YAML files.
Added dashboards and visualizations.
Added system test cases for the data stream.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

How to test this PR locally

Clone integrations repo.
Install elastic-package locally.
Start elastic stack using elastic-package.
Move to integrations/packages/mongodb_atlas) directory.
Run the following command to run tests. elastic-package test

Screenshots

elasticmachine · 2024-04-23T17:36:29Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

shmsr · 2024-05-03T05:07:06Z

packages/mongodb_atlas/_dev/build/docs/README.md

+- `mongod_database`: This data stream collects a running log of events, including entries such as incoming connections, commands run, and issues encountered. Generally, database log messages are useful for diagnosing issues, monitoring your deployment, and tuning performance.
+- `process`: This data stream collects host metrics per process for all the hosts of the specified group. Metrics like measurements for the host, such as CPU usage, number of I/O operations and memory are available on this data stream.


Suggested change

- `mongod_database`: This data stream collects a running log of events, including entries such as incoming connections, commands run, and issues encountered. Generally, database log messages are useful for diagnosing issues, monitoring your deployment, and tuning performance.

- `process`: This data stream collects host metrics per process for all the hosts of the specified group. Metrics like measurements for the host, such as CPU usage, number of I/O operations and memory are available on this data stream.

- `mongod_database`: This data stream collects a running log of events, including entries such as incoming connections, commands run, and issues encountered. Generally, database log messages are useful for diagnosing issues, monitoring your deployment, and tuning performance.

- `process`: This data stream collects host metrics per process for all the hosts in the specified group. Metrics like measurements for the host, such as CPU usage, number of I/O operations, and memory usage are available in this data stream.

shmsr · 2024-05-03T05:12:33Z

packages/mongodb_atlas/_dev/build/docs/README.md

@@ -101,6 +100,13 @@ This is the `mongod_database` data stream. This datastream collects a running lo

 ## Metrics reference

+### Hardware
+This data stream collects hardware and status metrics per process of the specified group. Metrics like measurements for the hardware and status, such as CPU usage and JVM memory usage are available on this data stream.


Suggested change

This data stream collects hardware and status metrics per process of the specified group. Metrics like measurements for the hardware and status, such as CPU usage and JVM memory usage are available on this data stream.

This data stream collects hardware and status metrics for each process in the specified group. It includes measurements such as CPU usage, memory consumption, JVM memory usage, disk usage, etc.

shmsr · 2024-05-03T05:13:09Z

packages/mongodb_atlas/changelog.yml

@@ -1,4 +1,9 @@
 # newer versions go on top
+- version: "0.0.4"
+  changes:
+    - description: MongoDB Atlas integration package with "hardware" data stream.


Suggested change

- description: MongoDB Atlas integration package with "hardware" data stream.

- description: Add "hardware" data stream to MongoDB Atlas package.

shmsr · 2024-05-03T05:18:59Z

packages/mongodb_atlas/data_stream/hardware/elasticsearch/ingest_pipeline/default.yml

+      value: mongodb_atlas
+  - set:
+      field: event.category
+      value: ["driver"]


Suggested change

value: ["driver"]

value: ["driver"]

shouldn't this category be "database"?

https://www.elastic.co/guide/en/ecs/current/ecs-allowed-values-event-category.html#ecs-event-category-database

accordingly find the correct event.type too.

shmsr · 2024-05-03T05:24:27Z

packages/mongodb_atlas/data_stream/hardware/elasticsearch/ingest_pipeline/default.yml

+  - rename:
+      field: status.JVM_MAX_MEMORY
+      target_field: mongodb_atlas.hardware.jvm.memory.heap.available.mb
+      ignore_missing: true
+  - rename:
+      field: status.JVM_CURRENT_MEMORY
+      target_field: mongodb_atlas.hardware.jvm.memory.heap.used.mb
+      ignore_missing: true


Don't you thinking we should stick to mongodb_atlas.hardware.jvm.max.memory and mongodb_atlas.hardware.jvm.current.memory? I know by definition it is correct but what's the problem to keep the target fields similar to source fields?

Also in the docs there's no mention of the unit. Probably unit could be anything. We shouldn't assume that it is always mb. Also, there's difference between {m,M}{b,B} i.e., megabits and megabytes. We should avoid this.

Problem with mongodb_atlas.hardware.jvm.max.memory is that even though it is similar to source field it's name is very different than definition, the metric it self collects Total amount of available memory in the JVM heap. so including max in its name and excluding available may lead to confusion for users. mongodb_atlas.hardware.jvm.current.memory does not have the same problem so we can use something like mongodb_atlas.hardware.jvm.memory.heap.current.mb. The main issue with keeping field names similar to source field is that we have our own approach for field names such as using suffix and grouping similar fields together, which results in to distinction between the source field name and our field name.

We receive the unit type of all the fields in raw response we get from MongoDB Atlas. So we have valid source to identify that these fields are megabytes. but I understand your concern with Mb & MB, to solve that we can add Megabytes in description of the fields.

Let me know your thoughts.

I agree with your first point the the definition does not exactly match with the field name. We can keep as it is then. But we have to remove the unit.

We receive the unit type of all the fields in raw response we get from MongoDB Atlas.

But it supports other types too. Did you test it in a setup where GB's of memory is available to JVM? What if then the response has gb as the unit? I could find the unit neither in the field not in the definition. My +1 would be to remove the unit.

but I understand your concern with Mb & MB, to solve that we can add Megabytes in the description of the fields.

Yeah, cool. Thanks!

But it supports other types too. Did you test it in a setup where GB's of memory is available to JVM?

i agree with your point. let me remove the unit type.

Hey @shmsr i checked the behavior of the metric, Even if size of data increases the metrics is still pretionted in its specified unit. below is an example.

Thanks for checking.

shmsr · 2024-05-03T05:37:29Z

packages/mongodb_atlas/data_stream/hardware/fields/fields.yml

+          description: Average rate of page faults on this process per second over the selected sample period.
+    - name: process_id
+      type: keyword
+      description: Combination of hostname and Internet Assigned Numbers Authority (IANA) port that serves the MongoDB process.


I think we should not use IANA port in the description. Pretty sure, people use it on ports that are not IANA port i.e., 27017.

Bettter use something like: "MongoDB process port"

shmsr · 2024-05-03T05:41:24Z

packages/mongodb_atlas/data_stream/hardware/fields/fields.yml

+                          metric_type: gauge
+                          unit: percent
+                          description: Percentage of time that the CPU spent servicing user calls for the search process.
+        - name: jvm


Shouldn't this be under status group? page_faults too?

I can create one group for status under hardware.

shmsr · 2024-05-03T05:42:02Z

packages/mongodb_atlas/data_stream/hardware/fields/fields.yml

+                    - name: available.mb
+                      type: long
+                      metric_type: counter
+                      description: Total amount of available memory in the JVM heap.
+                    - name: used.mb
+                      type: long
+                      metric_type: gauge
+                      description: Amount of memory that the JVM heap is currently using.


units not mentioned here.

We dont have mb as supported unit currently.

Probably it should be byte.

As per this, they call it "byte value".

See if you can verify? I did not double-check.

I think setting unit to byte will create confusion for users, since if the real data is not in bytes, but in read me it will show as byte. Also I can not find any solid documentation which would define unit type byte and if it can be used for mb.

Okay, cool.

shmsr · 2024-05-03T05:43:03Z

packages/mongodb_atlas/data_stream/hardware/manifest.yml

@@ -0,0 +1,54 @@
+title: Collect Hardware metrics from MongoDB Atlas
+type: logs


I thought it was metrics?

Yes, The data we are collecting is metrics, but we are using CEL input here, which falls under filebeat. So unfortunately the type here can not be changed.

shmsr · 2024-05-03T05:44:53Z

packages/mongodb_atlas/data_stream/hardware/manifest.yml

+        default: 10m
+        multi: false
+        required: true
+        show_user: false


Period should ideally shown to user by default?

muthu-mps · 2024-05-06T13:41:44Z

packages/mongodb_atlas/data_stream/hardware/fields/fields.yml

+  type: group
+  fields:
+    - name: group_id
+      description: Identifier for the project of the event.


Can we replace the description for the group_id. Unique identifier that identifies the project.

muthu-mps · 2024-05-06T13:41:55Z

packages/mongodb_atlas/_dev/build/docs/README.md


 Data streams:
+- `hardware`: This data stream collects all the Atlas Search hardware and status data series within the provided time range for one process in the specified project.


Can you help me understand the term Atlas Search hardware and status data series. While going through the Atlas documentation it has search metrics and hardware metrics. Are we referring to the search metrics and hardware metrics?

No, we are not referring to search and hardware metrics here. Here is documentation which will help you understand the hardware and status metrics.

shmsr · 2024-05-08T07:18:54Z

@niraj-elastic Hey, let's fix the merge conflicts and rebase with main as other changes also came to this PR. Because of the unrelated changes multiple teams are tagged to this. Unassigning them for now to whom I can until this is fixed.

shmsr · 2024-05-08T09:42:13Z

Removed review requests for unrelated teams. Also, looks like you have addressed all the review comments. The changes look good.

elasticmachine · 2024-05-08T12:03:35Z

💚 Build Succeeded

Buildkite Build
Commit: f256f7e

History

💚 Build #11252 succeeded ad9d9f2
💔 Build #11248 failed 4e28497
💚 Build #10760 succeeded b80ebf2
💚 Build #10731 succeeded fe50fa7

cc @niraj-elastic

elastic-sonarqube · 2024-05-08T12:03:38Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
96.1% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

muthu-mps

LGTM!

elasticmachine · 2024-05-09T06:49:15Z

Package mongodb_atlas - 0.0.4 containing this change is available at https://epr.elastic.co/search?package=mongodb_atlas

add hardware datastream

565a6eb

niraj-elastic requested a review from a team as a code owner April 23, 2024 17:19

update changelog

fe50fa7

niraj-elastic self-assigned this Apr 23, 2024

niraj-elastic added the Integration:MongoDBAtlas label Apr 23, 2024

update fields names

b80ebf2

niraj-elastic requested review from milan-elastic and ishleenk17 April 24, 2024 11:29

milan-elastic approved these changes Apr 24, 2024

View reviewed changes

shmsr requested changes May 3, 2024

View reviewed changes

shmsr added the enhancement New feature or request label May 3, 2024

muthu-mps reviewed May 6, 2024

View reviewed changes

niraj-elastic requested review from a team as code owners May 8, 2024 06:59

niraj-elastic requested review from gizas and constanca-m May 8, 2024 06:59

shmsr removed request for gizas and constanca-m May 8, 2024 07:19

shmsr changed the title ~~[MongoDB Atlas] Hardware data stream~~ [MongoDB Atlas] Add Hardware data stream May 8, 2024

shmsr removed request for a team May 8, 2024 09:41

shmsr approved these changes May 8, 2024

View reviewed changes

revert unwanted changes

f256f7e

niraj-elastic force-pushed the package_mongodb_atlas_hardware branch from ad9d9f2 to f256f7e Compare May 8, 2024 11:45

niraj-elastic requested a review from muthu-mps May 8, 2024 11:50

muthu-mps approved these changes May 9, 2024

View reviewed changes

shmsr merged commit 47b76fa into elastic:main May 9, 2024
5 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[MongoDB Atlas] Add Hardware data stream #9689

[MongoDB Atlas] Add Hardware data stream #9689

niraj-elastic commented Apr 23, 2024 •

edited by shmsr

elasticmachine commented Apr 23, 2024

shmsr May 3, 2024

shmsr May 3, 2024

shmsr May 3, 2024

shmsr May 3, 2024

shmsr May 3, 2024

niraj-elastic May 3, 2024

shmsr May 6, 2024

niraj-elastic May 6, 2024

niraj-elastic May 7, 2024

shmsr May 8, 2024

shmsr May 3, 2024

shmsr May 3, 2024

niraj-elastic May 6, 2024

shmsr May 6, 2024

shmsr May 3, 2024

niraj-elastic May 6, 2024

shmsr May 6, 2024

niraj-elastic May 6, 2024

shmsr May 6, 2024

shmsr May 3, 2024

niraj-elastic May 6, 2024

shmsr May 6, 2024

shmsr May 3, 2024

muthu-mps May 6, 2024

muthu-mps May 6, 2024

niraj-elastic May 7, 2024

shmsr commented May 8, 2024

shmsr commented May 8, 2024 •

edited

elasticmachine commented May 8, 2024

elastic-sonarqube bot commented May 8, 2024

muthu-mps left a comment

elasticmachine commented May 9, 2024

		- `mongod_database`: This data stream collects a running log of events, including entries such as incoming connections, commands run, and issues encountered. Generally, database log messages are useful for diagnosing issues, monitoring your deployment, and tuning performance.
		- `process`: This data stream collects host metrics per process for all the hosts of the specified group. Metrics like measurements for the host, such as CPU usage, number of I/O operations and memory are available on this data stream.

	This data stream collects hardware and status metrics per process of the specified group. Metrics like measurements for the hardware and status, such as CPU usage and JVM memory usage are available on this data stream.
	This data stream collects hardware and status metrics for each process in the specified group. It includes measurements such as CPU usage, memory consumption, JVM memory usage, disk usage, etc.

	- description: MongoDB Atlas integration package with "hardware" data stream.
	- description: Add "hardware" data stream to MongoDB Atlas package.

		@@ -0,0 +1,54 @@
		title: Collect Hardware metrics from MongoDB Atlas
		type: logs


		Data streams:
		- `hardware`: This data stream collects all the Atlas Search hardware and status data series within the provided time range for one process in the specified project.

[MongoDB Atlas] Add Hardware data stream #9689

[MongoDB Atlas] Add Hardware data stream #9689

Conversation

niraj-elastic commented Apr 23, 2024 • edited by shmsr

What does this PR do?

Checklist

How to test this PR locally

Screenshots

elasticmachine commented Apr 23, 2024

🚀 Benchmarks report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shmsr commented May 8, 2024

shmsr commented May 8, 2024 • edited

elasticmachine commented May 8, 2024

💚 Build Succeeded

History

elastic-sonarqube bot commented May 8, 2024

Quality Gate passed

muthu-mps left a comment

Choose a reason for hiding this comment

elasticmachine commented May 9, 2024

niraj-elastic commented Apr 23, 2024 •

edited by shmsr

shmsr commented May 8, 2024 •

edited