[AWS][RDS] Add metric type to fields #6094

constanca-m · 2023-05-04T14:53:11Z

What does this PR do?

Add metric type to fields of RDS data stream, necessary to support TSDB in the future.

Details

Gauge and counter fields were defined based on:

The metrics that represent a specific value in time, without consideration for the previous one, are gauge.
If a metric keeps increment and decrementing, it is a gauge.
If a metric is cumulative, then it is a counter - but it can only increment.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

How to test this PR locally

Refer to #6078

Related issues

Relates to [AWS][RDS] Enable TSDB support #6078

Signed-off-by: constanca-m <constanca.manteigas@elastic.co>

elasticmachine · 2023-05-04T15:42:41Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-06-01T16:22:29.472+0000
Duration: 55 min 45 sec

Test stats 🧪

Test	Results
Failed	0
Passed	188
Skipped	4
Total	192

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

elasticmachine · 2023-05-04T15:42:49Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`15/15`)	💚
Files	93.75% (`15/16`)	❕
Classes	93.75% (`15/16`)	❕
Methods	86.131% (`236/274`)	❕
Lines	85.925% (`7387/8597`)	❕
Conditionals	100.0% (`0/0`)	💚

tetianakravchenko · 2023-05-17T13:01:42Z

packages/aws/data_stream/rds/fields/fields.yml

@@ -25,321 +25,422 @@
    - name: rds
      type: group
      fields:
-        - name: cpu.total.pct


this format is not working? any specific reason to change it?

The format work, but the file has many fields and it was easier to group them and made changes this way @tetianakravchenko

tetianakravchenko · 2023-06-01T14:42:43Z

packages/aws/data_stream/rds/fields/fields.yml

+          type: group
+          fields:
+            - name: replicated_write_io.bytes
+              metric_type: counter


so this value is not an aggregated value during some period of time, similar to other metrics? I thought cloudwatch always provides aggregated values

the same question regarding the data_transfer.bytes

The problem with these metrics is that are only considered for a specific time window, like 15 minutes. So if we receive 2 documents, there is no guarantee that the most recent document will have a higher value for that field than the other. @tetianakravchenko

And since we only receive a document after that time window (I think), there is no point in using counter since we never have values that relate to each other @tetianakravchenko

shouldn't it then be a gauge? instead of counter? 🤔

Yes, I was making a case for gauge, I thought I had set it as such. But now I am confused on why I set this metric as a counter in the first place 😕

I fixed it @tetianakravchenko to be a gauge

tetianakravchenko · 2023-06-01T15:14:40Z

packages/aws/data_stream/rds/fields/fields.yml

+              description: |
+                The amount of network throughput both received from and transmitted to clients by each instance in the Aurora MySQL DB cluster, in bytes per second.
+            - name: network_receive
+              metric_type: gauge


if it is not an aggregated data, should it be counter then?

then same regarding network_transmit

Same case as the other. Practical example of a visualization for one specific instance:

Signed-off-by: constanca-m <constanca.manteigas@elastic.co>

tetianakravchenko · 2023-06-02T07:28:58Z

packages/aws/data_stream/rds/fields/fields.yml

          type: long
          format: bytes
          description: |
            The amount of available storage space.
        - name: maximum_used_transaction_ids
+          metric_type: counter


could you please double-check if this should be a counter? sounds reasonable, based on the description, but just in case

tetianakravchenko · 2023-06-02T07:29:30Z

packages/aws/data_stream/rds/fields/fields.yml

          type: long
          description: |
            The percentage of requests that are served by the Resultset cache.
        - name: engine_uptime.sec
+          metric_type: counter


the same here: could you please double-check if this should be a counter? sounds reasonable, but just in case

Yes, I checked yesterday and I believe both are counter. @tetianakravchenko

Example of this metric (this visualizations shows the difference between the last value and the new value):

hmm, strange to see this spike between 13:30 and 14:30, if this metric is suppose to monotonically increase, no? and why later value is not increasing?

They are increasing @tetianakravchenko The difference between the previous value and the current value is always >0. I expressed myself wrongly, sorry for that. We see a huge spike because we received no values for some time (see the same visualization, this time including the empty rows):

So when we received the value at 14:xxh, the last one we had was the one from 13:xx, so there was a bigger difference.

elasticmachine · 2023-06-02T10:44:16Z

Package aws - 1.38.1 containing this change is available at https://epr.elastic.co/search?package=aws

* Add metric type. Signed-off-by: constanca-m <constanca.manteigas@elastic.co>

constanca-m and others added 3 commits May 4, 2023 16:48

Add metric type.

a081e2d

Signed-off-by: constanca-m <constanca.manteigas@elastic.co>

Merge branch 'elastic:main' into rds-add-metrics

d0ff51c

Update changelog.

3944795

Signed-off-by: constanca-m <constanca.manteigas@elastic.co>

constanca-m added the Integration:AWS label May 4, 2023

constanca-m requested review from tetianakravchenko and agithomas May 4, 2023 14:53

constanca-m self-assigned this May 4, 2023

constanca-m requested a review from a team as a code owner May 4, 2023 14:53

Update changelog.

36c2ea8

Signed-off-by: constanca-m <constanca.manteigas@elastic.co>

constanca-m mentioned this pull request May 4, 2023

[AWS][RDS] Enable TSDB support #6078

Closed

6 tasks

tetianakravchenko reviewed May 17, 2023

View reviewed changes

kaiyan-sheng approved these changes May 17, 2023

View reviewed changes

tetianakravchenko reviewed Jun 1, 2023

View reviewed changes

constanca-m and others added 5 commits June 1, 2023 18:07

Fix metric type

c09201d

Signed-off-by: constanca-m <constanca.manteigas@elastic.co>

Update md file

4781075

Signed-off-by: constanca-m <constanca.manteigas@elastic.co>

Fix conflicts.

60a6dcc

Signed-off-by: constanca-m <constanca.manteigas@elastic.co>

Merge branch 'main' into rds-add-metrics

61cedd4

Update md file.

a330d66

Signed-off-by: constanca-m <constanca.manteigas@elastic.co>

tetianakravchenko reviewed Jun 2, 2023

View reviewed changes

tetianakravchenko approved these changes Jun 2, 2023

View reviewed changes

constanca-m merged commit cdb74ef into elastic:main Jun 2, 2023
3 checks passed

constanca-m deleted the rds-add-metrics branch June 2, 2023 08:44

sodhikirti07 pushed a commit that referenced this pull request Jun 15, 2023

[AWS][RDS] Add metric type to fields (#6094)

3b3d65d

* Add metric type. Signed-off-by: constanca-m <constanca.manteigas@elastic.co>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[AWS][RDS] Add metric type to fields #6094

[AWS][RDS] Add metric type to fields #6094

constanca-m commented May 4, 2023

elasticmachine commented May 4, 2023 •

edited

Build stats

Test stats 🧪

elasticmachine commented May 4, 2023 •

edited

tetianakravchenko May 17, 2023

constanca-m May 17, 2023

tetianakravchenko Jun 1, 2023

constanca-m Jun 1, 2023

constanca-m Jun 1, 2023

tetianakravchenko Jun 1, 2023

constanca-m Jun 1, 2023

constanca-m Jun 1, 2023

tetianakravchenko Jun 1, 2023

constanca-m Jun 1, 2023

tetianakravchenko Jun 2, 2023 •

edited

tetianakravchenko Jun 2, 2023

constanca-m Jun 2, 2023

constanca-m Jun 2, 2023 •

edited

tetianakravchenko Jun 2, 2023

constanca-m Jun 2, 2023 •

edited

elasticmachine commented Jun 2, 2023

[AWS][RDS] Add metric type to fields #6094

[AWS][RDS] Add metric type to fields #6094

Conversation

constanca-m commented May 4, 2023

What does this PR do?

Details

Checklist

How to test this PR locally

Related issues

elasticmachine commented May 4, 2023 • edited

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

elasticmachine commented May 4, 2023 • edited

🌐 Coverage report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tetianakravchenko Jun 2, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

constanca-m Jun 2, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

constanca-m Jun 2, 2023 • edited

Choose a reason for hiding this comment

elasticmachine commented Jun 2, 2023

elasticmachine commented May 4, 2023 •

edited

elasticmachine commented May 4, 2023 •

edited

tetianakravchenko Jun 2, 2023 •

edited

constanca-m Jun 2, 2023 •

edited

constanca-m Jun 2, 2023 •

edited