[Azure] [Container-service] Add dimension and metric_type metadata to the container_service datastream #7139

tetianakravchenko · 2023-07-25T14:34:42Z

What does this PR do?

remove duplicated fields
instead of using wildcard in the field name - use predefined field names
- for azure.dimensions - dimension names are predefined - https://learn.microsoft.com/en-us/azure/azure-monitor/reference/supported-metrics/microsoft-containerservice-managedclusters-metrics
- list of metrics - https://learn.microsoft.com/en-us/azure/azure-monitor/reference/supported-metrics/microsoft-containerservice-managedclusters-metrics
used fields for dimensions:
- cloud.region
- azure.resource.id - this field includes azure.subscription_id, azure.resource.group, azure.resource.name, format:
/subscriptions/{azure.subscription_id}/resourceGroups/{azure.resource.group}/providers/Microsoft.ContainerInstance/containerGroups/{azure.resource.name}
- azure.dimensions.node
- azure.dimensions.status
- azure.dimensions.condition
- azure.dimensions.pod
- azure.timegrain
metrics - use predefined metrics - since in the configuration is defined list of metrics

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

[ ]

How to test this PR locally

Related issues

Related issue with this data_strem: [Azure] Metrics are not grouped #7027 (comment) - some duplicated documents

Screenshots

…stream Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co>

Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co>

… using wildcard Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co>

Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co>

packages/azure_metrics/data_stream/container_service/fields/fields.yml

elasticmachine · 2023-07-25T14:49:49Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-08-08T13:58:18.496+0000
Duration: 15 min 34 sec

Test stats 🧪

Test	Results
Failed	0
Passed	134
Skipped	0
Total	134

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

elasticmachine · 2023-07-25T14:50:01Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`3/3`)	💚
Files	100.0% (`3/3`)	💚
Classes	100.0% (`3/3`)	💚
Methods	73.333% (`22/30`)	👍 7.018
Lines	100.0% (`21/21`)	💚 4.0
Conditionals	100.0% (`0/0`)	💚

packages/azure_metrics/data_stream/container_service/fields/package-fields.yml

Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co>

mlunadia · 2023-07-28T11:44:08Z

@bturquet can you help us getting these reviewed? We have a long queue of Azure packages waiting on it. Merci

agithomas · 2023-07-31T08:58:19Z

packages/azure_metrics/data_stream/container_service/fields/package-fields.yml

+          type: keyword
+          dimension: true
+          description: Node name
+        - name: status


Is status really required? Can a pod/container be in two different status in a given time series?

yes, you are right, available values for this field are: True (or true) and False(or false). I will remove dimension from this

agithomas · 2023-07-31T08:59:58Z

packages/azure_metrics/data_stream/container_service/fields/package-fields.yml

+        - name: condition
+          type: keyword
+          dimension: true
+          description: The container name


Please check the description. Does condition and status field. Both has same description. Is it correct?

What value goes under condition to promote it as a dimension field?

possible values of condition:

Note: that those conditions are for the same node

agithomas · 2023-07-31T09:12:10Z

Should application.id and namespace be considered as dimensions?

tetianakravchenko · 2023-08-02T14:25:59Z

Should application.id and namespace be considered as dimensions?

@agithomas
I wasn't able to get application_id field for this datastream. It is also not available in sample_event

@zmoog could it be that this field was just copied from other datastreams, but it is not suppose to be here?

regarding the namespace field - since all documents for this datastream have the same namespace - Microsoft.ContainerService/managedClusters I haven't added it

zmoog · 2023-08-02T15:07:32Z

Should application.id and namespace be considered as dimensions?

I wasn't able to get application_id field for this datastream. It is also not available in sample_event

@zmoog could it be that this field was just copied from other datastreams, but it is not suppose to be here?

This field mapping is included in all metrics data stream, including the Billing and Application Insights and Application State.

I see the Application Insights' sample_event.json file has a application_id field, it is set by the metricset, and it makes sense since it's an APM-esque service.

There's a good chance azure.application_id comes from a copy & paste. Should I create an issue to remove this mapping?

…on: true for the azure.dimwnsions.status field; fix changelog merge conflict Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co>

Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co>

tetianakravchenko · 2023-08-09T08:31:12Z

@agithomas I've added missing dimensions, can you please have another look on this PR?

agithomas

LGTM!

elasticmachine · 2023-08-10T14:07:28Z

Package azure_metrics - 1.0.19 containing this change is available at https://epr.elastic.co/search?package=azure_metrics

… the container_service datastream (#7139) * Add dimension and metric_type metadata to the container_service data_stream Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co> * rename dimensions.* -> dimensions Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co> * add list of metrics to the azure.container_service object, instead of using wildcard Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co> * fix pr link and changelog Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co> * revert version Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co> * remove metrics.* fields; add agent.id as a dimension Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co> * push fields documentation changes Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co> --------- Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co>

tetianakravchenko added 3 commits July 24, 2023 18:31

Add dimension and metric_type metadata to the container_service data_…

7a480d1

…stream Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co>

rename dimensions.* -> dimensions

c9db8f1

Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co>

add list of metrics to the azure.container_service object, instead of…

3dc8f6a

… using wildcard Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co>

tetianakravchenko requested a review from a team as a code owner July 25, 2023 14:34

tetianakravchenko added 2 commits July 25, 2023 16:35

fix pr link and changelog

0701bac

Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co>

revert version

3caeb30

Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co>

tetianakravchenko commented Jul 25, 2023

View reviewed changes

packages/azure_metrics/data_stream/container_service/fields/fields.yml Show resolved Hide resolved

tetianakravchenko requested review from zmoog and agithomas July 25, 2023 14:40

tetianakravchenko changed the title ~~[Azure] [Container-service]~~ [Azure] [Container-service] Add dimension and metric_type metadata to the container_service datastream Jul 25, 2023

tetianakravchenko commented Jul 26, 2023

View reviewed changes

packages/azure_metrics/data_stream/container_service/fields/package-fields.yml Show resolved Hide resolved

tetianakravchenko mentioned this pull request Jul 26, 2023

[Azure] TSDB enablement - track all metrics data streams #7140

Closed

31 tasks

tetianakravchenko commented Jul 26, 2023

View reviewed changes

packages/azure_metrics/data_stream/container_service/fields/package-fields.yml Outdated Show resolved Hide resolved

remove metrics.* fields; add agent.id as a dimension

0beff2c

Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co>

zmoog approved these changes Jul 28, 2023

View reviewed changes

agithomas reviewed Jul 31, 2023

View reviewed changes

tommyers-elastic mentioned this pull request Jul 31, 2023

[Azure] [container_service] Review list of dimensions #7161

Closed

andrewkroh added the Integration:Azure label Aug 1, 2023

tetianakravchenko added 3 commits August 2, 2023 18:43

Change descriptions for the azure.dimensions.* fields; remove dimensi…

2ce9f5c

…on: true for the azure.dimwnsions.status field; fix changelog merge conflict Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co>

push fields documentation changes

ec3ccf5

Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co>

merge in main branch; add namespace and phase azure.dimension

a0e2d02

Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co>

agithomas approved these changes Aug 10, 2023

View reviewed changes

tetianakravchenko merged commit 0c88f39 into elastic:main Aug 10, 2023
4 checks passed

tetianakravchenko deleted the azure-container-service-tsdb branch August 10, 2023 13:44

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Azure] [Container-service] Add dimension and metric_type metadata to the container_service datastream #7139

[Azure] [Container-service] Add dimension and metric_type metadata to the container_service datastream #7139

tetianakravchenko commented Jul 25, 2023 •

edited

Loading

elasticmachine commented Jul 25, 2023 •

edited

Loading

Build stats

Test stats 🧪

elasticmachine commented Jul 25, 2023 •

edited

Loading

mlunadia commented Jul 28, 2023

agithomas Jul 31, 2023

tetianakravchenko Aug 2, 2023

agithomas Jul 31, 2023

agithomas Jul 31, 2023

tetianakravchenko Aug 2, 2023

agithomas commented Jul 31, 2023

tetianakravchenko commented Aug 2, 2023 •

edited

Loading

zmoog commented Aug 2, 2023

tetianakravchenko commented Aug 9, 2023

agithomas left a comment

elasticmachine commented Aug 10, 2023

[Azure] [Container-service] Add dimension and metric_type metadata to the container_service datastream #7139

[Azure] [Container-service] Add dimension and metric_type metadata to the container_service datastream #7139

Conversation

tetianakravchenko commented Jul 25, 2023 • edited Loading

What does this PR do?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

elasticmachine commented Jul 25, 2023 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

elasticmachine commented Jul 25, 2023 • edited Loading

🌐 Coverage report

mlunadia commented Jul 28, 2023

agithomas Jul 31, 2023

Choose a reason for hiding this comment

tetianakravchenko Aug 2, 2023

Choose a reason for hiding this comment

agithomas Jul 31, 2023

Choose a reason for hiding this comment

agithomas Jul 31, 2023

Choose a reason for hiding this comment

tetianakravchenko Aug 2, 2023

Choose a reason for hiding this comment

agithomas commented Jul 31, 2023

tetianakravchenko commented Aug 2, 2023 • edited Loading

zmoog commented Aug 2, 2023

tetianakravchenko commented Aug 9, 2023

agithomas left a comment

Choose a reason for hiding this comment

elasticmachine commented Aug 10, 2023

tetianakravchenko commented Jul 25, 2023 •

edited

Loading

elasticmachine commented Jul 25, 2023 •

edited

Loading

elasticmachine commented Jul 25, 2023 •

edited

Loading

tetianakravchenko commented Aug 2, 2023 •

edited

Loading