add_cloud_metadata - Retrieve AKS cluster name and id #37685

MichaelKatsoulis · 2024-01-22T12:02:55Z

Proposed commit message

WHAT: Enhance add_cloud_metadata processor with orchestrator.cluster.name and orchestrator.cluster.id for AKS clusters.
WHY: It allows filtering of events based on the AKS cluster the vm belongs to.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

Create an AKS cluster
Run Metricbeat or Filebeat with sleep command and correct values of following additional env vars

- name: TENANT_ID
 value: ""
- name: CLIENT_ID
 value: ""
- name: CLIENT_SECRET
 value: ""

In order to acquire a CLIENT_SECRET and CLIENT_ID follow the app_registration instructions.

Build this pr's version of metricbeat/filebeat with GOOS=linux GOARCH=amd64 go build
Copy the metricbeat/filebeat binary in metricbeat/filebeat pod with kubectl cp ./metricbeat metricbeat-9lkhf:/usr/share/metricbeat/ -n kube-system
Exec in the pod and run ./metricbeat -c /etc/metricbeat.yml -e
Check in Kibana the fields orchestrator.cluster.name and orchestrator.cluster.id being populated

Related issues

Closes Investigate if it is possible to set orchestrator fields from Cloud provider kubernetes metadata #33081

Use cases

Screenshots

Logs

mergify · 2024-01-22T12:03:40Z

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @MichaelKatsoulis? 🙏.
For such, you'll need to label your PR with:

The upcoming major version of the Elastic Stack
The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

elasticmachine · 2024-01-22T14:20:48Z

💔 Build Failed

Expand to view the summary

Build stats

Duration: 137 min 19 sec

Pipeline error

❕ Flaky test report

No test was executed to be analysed.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

elasticmachine · 2024-01-24T15:17:54Z

❕ Build Aborted

Expand to view the summary

Build stats

Duration: 36 min 41 sec

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

elasticmachine · 2024-01-24T17:16:16Z

💔 Tests Failed

Expand to view the summary

Build stats

Start Time: 2024-01-24T15:09:11.210+0000
Duration: 127 min 49 sec

Test stats 🧪

Test	Results
Failed	2
Passed	28763
Skipped	1998
Total	30763

Test errors

Expand to view the tests failures

`Build&Test / libbeat-unitTest / TestRetrieveAzureMetadata – github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata`

Expand to view the error details

 Failed

Expand to view the stacktrace

 === RUN   TestRetrieveAzureMetadata
    provider_azure_vm_test.go:188: 
        	Error Trace:	/var/lib/jenkins/workspace/PR-37685-3-5434ad54-fc27-4ff7-93d7-63eaf703320a/src/github.com/elastic/beats/libbeat/processors/add_cloud_metadata/provider_azure_vm_test.go:188
        	Error:      	Not equal: 
        	            	expected: {"cloud":{"account":{"id":"0e073ec1-c22f-4488-adde-da35ed609ccd"},"instance":{"id":"220e2b43-0913-492f-ada0-b6b2795bdb9b","name":"aks-agentpool-12628255-vmss_2"},"machine":{"type":"Standard_DS2_v2"},"provider":"azure","region":"eastus","resourceGroup":{"name":"MC_myname_group_myname_eastus"},"service":{"name":"Virtual Machines"}},"orchestrator":{"cluster":{"id":"testcluster1Id","name":"testcluster1Name"}}}
        	            	actual  : {"cloud":{"account":{"id":"0e073ec1-c22f-4488-adde-da35ed609ccd"},"instance":{"id":"220e2b43-0913-492f-ada0-b6b2795bdb9b","name":"aks-agentpool-12628255-vmss_2"},"machine":{"type":"Standard_DS2_v2"},"provider":"azure","region":"eastus","resourceGroup":{"name":"MC_myname_group_myname_eastus"},"service":{"name":"Virtual Machines"}}}
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -1,2 +1,2 @@
        	            	-(mapstr.M) (len=2) {
        	            	+(mapstr.M) (len=1) {
        	            	  (string) (len=5) "cloud": (mapstr.M) (len=7) {
        	            	@@ -20,8 +20,2 @@
        	            	   }
        	            	- },
        	            	- (string) (len=12) "orchestrator": (mapstr.M) (len=1) {
        	            	-  (string) (len=7) "cluster": (mapstr.M) (len=2) {
        	            	-   (string) (len=2) "id": (string) (len=14) "testcluster1Id",
        	            	-   (string) (len=4) "name": (string) (len=16) "testcluster1Name"
        	            	-  }
        	            	  }
        	Test:       	TestRetrieveAzureMetadata
--- FAIL: TestRetrieveAzureMetadata (1.94s)

`Build&Test / libbeat-goIntegTest / TestRetrieveAzureMetadata – github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata`

Expand to view the error details

 Failed

Expand to view the stacktrace

 === RUN   TestRetrieveAzureMetadata
    provider_azure_vm_test.go:188: 
        	Error Trace:	/var/lib/jenkins/workspace/PR-37685-3-96180cfd-f1b8-42b2-bf0c-7fe3fac46860/src/github.com/elastic/beats/libbeat/processors/add_cloud_metadata/provider_azure_vm_test.go:188
        	Error:      	Not equal: 
        	            	expected: {"cloud":{"account":{"id":"0e073ec1-c22f-4488-adde-da35ed609ccd"},"instance":{"id":"220e2b43-0913-492f-ada0-b6b2795bdb9b","name":"aks-agentpool-12628255-vmss_2"},"machine":{"type":"Standard_DS2_v2"},"provider":"azure","region":"eastus","resourceGroup":{"name":"MC_myname_group_myname_eastus"},"service":{"name":"Virtual Machines"}},"orchestrator":{"cluster":{"id":"testcluster1Id","name":"testcluster1Name"}}}
        	            	actual  : {}
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -1,28 +1,2 @@
        	            	-(mapstr.M) (len=2) {
        	            	- (string) (len=5) "cloud": (mapstr.M) (len=7) {
        	            	-  (string) (len=7) "account": (mapstr.M) (len=1) {
        	            	-   (string) (len=2) "id": (string) (len=36) "0e073ec1-c22f-4488-adde-da35ed609ccd"
        	            	-  },
        	            	-  (string) (len=8) "instance": (mapstr.M) (len=2) {
        	            	-   (string) (len=2) "id": (string) (len=36) "220e2b43-0913-492f-ada0-b6b2795bdb9b",
        	            	-   (string) (len=4) "name": (string) (len=29) "aks-agentpool-12628255-vmss_2"
        	            	-  },
        	            	-  (string) (len=7) "machine": (mapstr.M) (len=1) {
        	            	-   (string) (len=4) "type": (string) (len=15) "Standard_DS2_v2"
        	            	-  },
        	            	-  (string) (len=8) "provider": (string) (len=5) "azure",
        	            	-  (string) (len=6) "region": (string) (len=6) "eastus",
        	            	-  (string) (len=13) "resourceGroup": (mapstr.M) (len=1) {
        	            	-   (string) (len=4) "name": (string) (len=29) "MC_myname_group_myname_eastus"
        	            	-  },
        	            	-  (string) (len=7) "service": (mapstr.M) (len=1) {
        	            	-   (string) (len=4) "name": (string) (len=16) "Virtual Machines"
        	            	-  }
        	            	- },
        	            	- (string) (len=12) "orchestrator": (mapstr.M) (len=1) {
        	            	-  (string) (len=7) "cluster": (mapstr.M) (len=2) {
        	            	-   (string) (len=2) "id": (string) (len=14) "testcluster1Id",
        	            	-   (string) (len=4) "name": (string) (len=16) "testcluster1Name"
        	            	-  }
        	            	- }
        	            	+(mapstr.M) {
        	            	 }
        	Test:       	TestRetrieveAzureMetadata
--- FAIL: TestRetrieveAzureMetadata (3.00s)

Steps errors

Expand to view the steps failures

`libbeat-unitTest - mage build unitTest`

Took 12 min 35 sec . View more details here
Description: mage build unitTest

`libbeat-unitTest - mage build unitTest`

Took 9 min 14 sec . View more details here
Description: mage build unitTest

`libbeat-unitTest - mage build unitTest`

Took 9 min 15 sec . View more details here
Description: mage build unitTest

`libbeat-goIntegTest - mage goIntegTest`

Took 16 min 40 sec . View more details here
Description: mage goIntegTest

`libbeat-goIntegTest - mage goIntegTest`

Took 12 min 32 sec . View more details here
Description: mage goIntegTest

`libbeat-goIntegTest - mage goIntegTest`

Took 11 min 21 sec . View more details here
Description: mage goIntegTest

`Error signal`

Took 0 min 0 sec . View more details here
Description: Error 'hudson.AbortException: script returned exit code 1'

🐛 Flaky test report

❕ There are test failures but not known flaky tests.

Expand to view the summary

Genuine test errors

💔 There are test failures but not known flaky tests, most likely a genuine test failure.

Name: Build&Test / libbeat-unitTest / TestRetrieveAzureMetadata – github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata
Name: Build&Test / libbeat-goIntegTest / TestRetrieveAzureMetadata – github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

elasticmachine · 2024-01-25T11:55:56Z

❕ Build Aborted

Expand to view the summary

Build stats

Start Time: 2024-01-25T10:06:38.271+0000
Duration: 109 min 10 sec

Test stats 🧪

Test	Results
Failed	0
Passed	3
Skipped	0
Total	3

Steps errors

Expand to view the steps failures

`Archive the artifacts`

Took 0 min 0 sec . View more details here

`Checks if running on a Unix-like node`

Took 0 min 0 sec . View more details here

`Archive the artifacts`

Took 0 min 0 sec . View more details here

`Checks if running on a Unix-like node`

Took 0 min 0 sec . View more details here

`Error signal`

Took 0 min 0 sec . View more details here
Description: Error 'org.jenkinsci.plugins.workflow.steps.FlowInterruptedException'

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

elasticmachine · 2024-01-25T11:57:43Z

❕ Build Aborted

Expand to view the summary

Build stats

Start Time: 2024-01-25T10:13:53.345+0000
Duration: 104 min 36 sec

Test stats 🧪

Test	Results
Failed	2
Passed	28329
Skipped	1998
Total	30329

Test errors

Expand to view the tests failures

`Build&Test / libbeat-unitTest / TestRetrieveAzureMetadata – github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata`

Expand to view the error details

 Failed

Expand to view the stacktrace

 === RUN   TestRetrieveAzureMetadata
    provider_azure_vm_test.go:189: 
        	Error Trace:	/var/lib/jenkins/workspace/PR-37685-5-247ed27c-a11b-4ed4-bd6e-36d84e2f7e1e/src/github.com/elastic/beats/libbeat/processors/add_cloud_metadata/provider_azure_vm_test.go:189
        	Error:      	Not equal: 
        	            	expected: {"cloud":{"account":{"id":"0e073ec1-c22f-4488-adde-da35ed609ccd"},"instance":{"id":"220e2b43-0913-492f-ada0-b6b2795bdb9b","name":"aks-agentpool-12628255-vmss_2"},"machine":{"type":"Standard_DS2_v2"},"provider":"azure","region":"eastus","resourceGroup":{"name":"MC_myname_group_myname_eastus"},"service":{"name":"Virtual Machines"}},"orchestrator":{"cluster":{"id":"testcluster1Id","name":"testcluster1Name"}}}
        	            	actual  : {"cloud":{"account":{"id":"0e073ec1-c22f-4488-adde-da35ed609ccd"},"instance":{"id":"220e2b43-0913-492f-ada0-b6b2795bdb9b","name":"aks-agentpool-12628255-vmss_2"},"machine":{"type":"Standard_DS2_v2"},"provider":"azure","region":"eastus","resourceGroup":{"name":"MC_myname_group_myname_eastus"},"service":{"name":"Virtual Machines"}}}
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -1,2 +1,2 @@
        	            	-(mapstr.M) (len=2) {
        	            	+(mapstr.M) (len=1) {
        	            	  (string) (len=5) "cloud": (mapstr.M) (len=7) {
        	            	@@ -20,8 +20,2 @@
        	            	   }
        	            	- },
        	            	- (string) (len=12) "orchestrator": (mapstr.M) (len=1) {
        	            	-  (string) (len=7) "cluster": (mapstr.M) (len=2) {
        	            	-   (string) (len=2) "id": (string) (len=14) "testcluster1Id",
        	            	-   (string) (len=4) "name": (string) (len=16) "testcluster1Name"
        	            	-  }
        	            	  }
        	Test:       	TestRetrieveAzureMetadata
--- FAIL: TestRetrieveAzureMetadata (1.95s)

`Build&Test / libbeat-goIntegTest / TestRetrieveAzureMetadata – github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata`

Expand to view the error details

 Failed

Expand to view the stacktrace

 === RUN   TestRetrieveAzureMetadata
    provider_azure_vm_test.go:189: 
        	Error Trace:	/var/lib/jenkins/workspace/PR-37685-5-3dfa0685-11f0-4630-81a2-bc3675d27050/src/github.com/elastic/beats/libbeat/processors/add_cloud_metadata/provider_azure_vm_test.go:189
        	Error:      	Not equal: 
        	            	expected: {"cloud":{"account":{"id":"0e073ec1-c22f-4488-adde-da35ed609ccd"},"instance":{"id":"220e2b43-0913-492f-ada0-b6b2795bdb9b","name":"aks-agentpool-12628255-vmss_2"},"machine":{"type":"Standard_DS2_v2"},"provider":"azure","region":"eastus","resourceGroup":{"name":"MC_myname_group_myname_eastus"},"service":{"name":"Virtual Machines"}},"orchestrator":{"cluster":{"id":"testcluster1Id","name":"testcluster1Name"}}}
        	            	actual  : {}
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -1,28 +1,2 @@
        	            	-(mapstr.M) (len=2) {
        	            	- (string) (len=5) "cloud": (mapstr.M) (len=7) {
        	            	-  (string) (len=7) "account": (mapstr.M) (len=1) {
        	            	-   (string) (len=2) "id": (string) (len=36) "0e073ec1-c22f-4488-adde-da35ed609ccd"
        	            	-  },
        	            	-  (string) (len=8) "instance": (mapstr.M) (len=2) {
        	            	-   (string) (len=2) "id": (string) (len=36) "220e2b43-0913-492f-ada0-b6b2795bdb9b",
        	            	-   (string) (len=4) "name": (string) (len=29) "aks-agentpool-12628255-vmss_2"
        	            	-  },
        	            	-  (string) (len=7) "machine": (mapstr.M) (len=1) {
        	            	-   (string) (len=4) "type": (string) (len=15) "Standard_DS2_v2"
        	            	-  },
        	            	-  (string) (len=8) "provider": (string) (len=5) "azure",
        	            	-  (string) (len=6) "region": (string) (len=6) "eastus",
        	            	-  (string) (len=13) "resourceGroup": (mapstr.M) (len=1) {
        	            	-   (string) (len=4) "name": (string) (len=29) "MC_myname_group_myname_eastus"
        	            	-  },
        	            	-  (string) (len=7) "service": (mapstr.M) (len=1) {
        	            	-   (string) (len=4) "name": (string) (len=16) "Virtual Machines"
        	            	-  }
        	            	- },
        	            	- (string) (len=12) "orchestrator": (mapstr.M) (len=1) {
        	            	-  (string) (len=7) "cluster": (mapstr.M) (len=2) {
        	            	-   (string) (len=2) "id": (string) (len=14) "testcluster1Id",
        	            	-   (string) (len=4) "name": (string) (len=16) "testcluster1Name"
        	            	-  }
        	            	- }
        	            	+(mapstr.M) {
        	            	 }
        	Test:       	TestRetrieveAzureMetadata
--- FAIL: TestRetrieveAzureMetadata (3.01s)

Steps errors

Expand to view the steps failures

`libbeat-unitTest - mage build unitTest`

Took 12 min 35 sec . View more details here
Description: mage build unitTest

`libbeat-unitTest - mage build unitTest`

Took 9 min 14 sec . View more details here
Description: mage build unitTest

`libbeat-unitTest - mage build unitTest`

Took 9 min 14 sec . View more details here
Description: mage build unitTest

`libbeat-goIntegTest - mage goIntegTest`

Took 16 min 39 sec . View more details here
Description: mage goIntegTest

`libbeat-goIntegTest - mage goIntegTest`

Took 12 min 45 sec . View more details here
Description: mage goIntegTest

`libbeat-goIntegTest - mage goIntegTest`

Took 12 min 44 sec . View more details here
Description: mage goIntegTest

`Error signal`

Took 0 min 0 sec . View more details here
Description: Error 'hudson.AbortException: script returned exit code 1'

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

elasticmachine · 2024-01-25T13:59:45Z

💔 Tests Failed

Expand to view the summary

Build stats

Start Time: 2024-01-25T11:50:34.518+0000
Duration: 129 min 53 sec

Test stats 🧪

Test	Results
Failed	2
Passed	28763
Skipped	1998
Total	30763

Test errors

Expand to view the tests failures

`Build&Test / libbeat-unitTest / TestRetrieveAzureMetadata – github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata`

Expand to view the error details

 Failed

Expand to view the stacktrace

 === RUN   TestRetrieveAzureMetadata
    provider_azure_vm_test.go:190: 
        	Error Trace:	/var/lib/jenkins/workspace/PR-37685-6-fe2b6713-3612-40d7-89c7-6537285a90ce/src/github.com/elastic/beats/libbeat/processors/add_cloud_metadata/provider_azure_vm_test.go:190
        	Error:      	Not equal: 
        	            	expected: {"cloud":{"account":{"id":"0e073ec1-c22f-4488-adde-da35ed609ccd"},"instance":{"id":"220e2b43-0913-492f-ada0-b6b2795bdb9b","name":"aks-agentpool-12628255-vmss_2"},"machine":{"type":"Standard_DS2_v2"},"provider":"azure","region":"eastus","resourceGroup":{"name":"MC_myname_group_myname_eastus"},"service":{"name":"Virtual Machines"}},"orchestrator":{"cluster":{"id":"testcluster1Id","name":"testcluster1Name"}}}
        	            	actual  : {"cloud":{"account":{"id":"0e073ec1-c22f-4488-adde-da35ed609ccd"},"instance":{"id":"220e2b43-0913-492f-ada0-b6b2795bdb9b","name":"aks-agentpool-12628255-vmss_2"},"machine":{"type":"Standard_DS2_v2"},"provider":"azure","region":"eastus","resourceGroup":{"name":"MC_myname_group_myname_eastus"},"service":{"name":"Virtual Machines"}}}
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -1,2 +1,2 @@
        	            	-(mapstr.M) (len=2) {
        	            	+(mapstr.M) (len=1) {
        	            	  (string) (len=5) "cloud": (mapstr.M) (len=7) {
        	            	@@ -20,8 +20,2 @@
        	            	   }
        	            	- },
        	            	- (string) (len=12) "orchestrator": (mapstr.M) (len=1) {
        	            	-  (string) (len=7) "cluster": (mapstr.M) (len=2) {
        	            	-   (string) (len=2) "id": (string) (len=14) "testcluster1Id",
        	            	-   (string) (len=4) "name": (string) (len=16) "testcluster1Name"
        	            	-  }
        	            	  }
        	Test:       	TestRetrieveAzureMetadata
--- FAIL: TestRetrieveAzureMetadata (2.09s)

`Build&Test / libbeat-goIntegTest / TestRetrieveAzureMetadata – github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata`

Expand to view the error details

 Failed

Expand to view the stacktrace

 === RUN   TestRetrieveAzureMetadata
    provider_azure_vm_test.go:190: 
        	Error Trace:	/var/lib/jenkins/workspace/PR-37685-6-2c8b8f66-131b-4cf9-86aa-795372e3a396/src/github.com/elastic/beats/libbeat/processors/add_cloud_metadata/provider_azure_vm_test.go:190
        	Error:      	Not equal: 
        	            	expected: {"cloud":{"account":{"id":"0e073ec1-c22f-4488-adde-da35ed609ccd"},"instance":{"id":"220e2b43-0913-492f-ada0-b6b2795bdb9b","name":"aks-agentpool-12628255-vmss_2"},"machine":{"type":"Standard_DS2_v2"},"provider":"azure","region":"eastus","resourceGroup":{"name":"MC_myname_group_myname_eastus"},"service":{"name":"Virtual Machines"}},"orchestrator":{"cluster":{"id":"testcluster1Id","name":"testcluster1Name"}}}
        	            	actual  : {}
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -1,28 +1,2 @@
        	            	-(mapstr.M) (len=2) {
        	            	- (string) (len=5) "cloud": (mapstr.M) (len=7) {
        	            	-  (string) (len=7) "account": (mapstr.M) (len=1) {
        	            	-   (string) (len=2) "id": (string) (len=36) "0e073ec1-c22f-4488-adde-da35ed609ccd"
        	            	-  },
        	            	-  (string) (len=8) "instance": (mapstr.M) (len=2) {
        	            	-   (string) (len=2) "id": (string) (len=36) "220e2b43-0913-492f-ada0-b6b2795bdb9b",
        	            	-   (string) (len=4) "name": (string) (len=29) "aks-agentpool-12628255-vmss_2"
        	            	-  },
        	            	-  (string) (len=7) "machine": (mapstr.M) (len=1) {
        	            	-   (string) (len=4) "type": (string) (len=15) "Standard_DS2_v2"
        	            	-  },
        	            	-  (string) (len=8) "provider": (string) (len=5) "azure",
        	            	-  (string) (len=6) "region": (string) (len=6) "eastus",
        	            	-  (string) (len=13) "resourceGroup": (mapstr.M) (len=1) {
        	            	-   (string) (len=4) "name": (string) (len=29) "MC_myname_group_myname_eastus"
        	            	-  },
        	            	-  (string) (len=7) "service": (mapstr.M) (len=1) {
        	            	-   (string) (len=4) "name": (string) (len=16) "Virtual Machines"
        	            	-  }
        	            	- },
        	            	- (string) (len=12) "orchestrator": (mapstr.M) (len=1) {
        	            	-  (string) (len=7) "cluster": (mapstr.M) (len=2) {
        	            	-   (string) (len=2) "id": (string) (len=14) "testcluster1Id",
        	            	-   (string) (len=4) "name": (string) (len=16) "testcluster1Name"
        	            	-  }
        	            	- }
        	            	+(mapstr.M) {
        	            	 }
        	Test:       	TestRetrieveAzureMetadata
--- FAIL: TestRetrieveAzureMetadata (3.06s)

Steps errors

Expand to view the steps failures

`libbeat-unitTest - mage build unitTest`

Took 11 min 24 sec . View more details here
Description: mage build unitTest

`libbeat-unitTest - mage build unitTest`

Took 9 min 13 sec . View more details here
Description: mage build unitTest

`libbeat-unitTest - mage build unitTest`

Took 9 min 17 sec . View more details here
Description: mage build unitTest

`libbeat-goIntegTest - mage goIntegTest`

Took 15 min 5 sec . View more details here
Description: mage goIntegTest

`libbeat-goIntegTest - mage goIntegTest`

Took 11 min 8 sec . View more details here
Description: mage goIntegTest

`libbeat-goIntegTest - mage goIntegTest`

Took 11 min 13 sec . View more details here
Description: mage goIntegTest

`Error signal`

Took 0 min 0 sec . View more details here
Description: Error 'hudson.AbortException: script returned exit code 1'

🐛 Flaky test report

❕ There are test failures but not known flaky tests.

Expand to view the summary

Genuine test errors

💔 There are test failures but not known flaky tests, most likely a genuine test failure.

Name: Build&Test / libbeat-unitTest / TestRetrieveAzureMetadata – github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata
Name: Build&Test / libbeat-goIntegTest / TestRetrieveAzureMetadata – github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

…e-id

elasticmachine · 2024-01-25T16:23:08Z

❕ Build Aborted

Expand to view the summary

Build stats

Duration: 98 min 10 sec

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

fearful-symmetry · 2024-03-05T17:03:59Z

libbeat/processors/add_cloud_metadata/provider_azure_vm.go

+		// hfetcher represents an http fetcher to retrieve metadata from azure metadata endpoint
+		hfetcher, err := newMetadataFetcher(config, "azure", azHeaders, metadataHost, azHttpSchema, azMetadataURI)
+		if err != nil {
+			return hfetcher, err


Can we add fmt.Errorf statements to the errors here? We put them in some of the later functions, so we should be consistent.

@fearful-symmetry Done

andrewkroh · 2024-03-07T14:07:50Z

libbeat/processors/add_cloud_metadata/_meta/fields.yml

@@ -53,4 +52,12 @@
      path: cloud.region
      migration: true

+    - name: azure.resourcegroup


Is this consistent with what we have been using in Fleet integrations? I think I recall azure.resource.group being used over there.

I am also a little concerned about this processor beginning to produce fields outside of the ECS cloud and orchestrator namespaces.

Downstream consumers (e.g Fleet integrations) should ideally be updated to have this field included in their mappings since it's not part of ECS. This affects almost all integrations since Agent includes add_cloud_metadata with most input types from Beats. (Imagine if we ran our CI on Azure for Fleet integration system testing, after this merges every integration's system tests would begin failing because this field was not defined in its mappings.)

The other concern is whether or not existing pipelines will safely handle the addition of this field. For example if an ingest pipeline is already using this field then does it safely handle the case where the field already exists.

The azure.resource.group , created by a pipeline? as I noticed is the resource group stripped. It only includes the group's name. The addition of the azure.resourcegroup.name field I did was to me a nice to have and definitely not the reason of this PR.
I understand that it then needs to be added in all the integrations because it comes from the processor.
Also it is confusing having two similar named fields. I chose the naming to be compliant with Otel.

I hear your concerns and I will remove this field as it beats the purpose of this PR which was to add the orchestrator ecs fields.

The azure.resource.group , created by a pipeline?

I'm not familiar with these integrations, but, yes, I see it being created by a grok in a few pipeline.

https://github.com/search?q=repo%3Aelastic%2Fintegrations+path%3A**%2F*.yml+%2Fazure%5C.resource%5C.%3Fgroup%2F&type=code

I understand that it then needs to be added in all the integrations because it comes from the processor.

This is a really general problem we have with Fleet integrations in that there is no easy or centralized way to account for the fields being added by the processors.

I hear your concerns and I will remove this field as it beats the purpose of this PR which was to add the orchestrator ecs fields.

👍

libbeat/processors/add_cloud_metadata/docs/add_cloud_metadata.asciidoc

libbeat/processors/add_cloud_metadata/provider_azure_vm.go

…asciidoc Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>

Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>

libbeat/processors/add_cloud_metadata/provider_azure_vm.go

andrewkroh · 2024-03-08T14:47:10Z

libbeat/processors/add_cloud_metadata/_meta/fields.yml

@@ -53,4 +52,12 @@
      path: cloud.region
      migration: true

+    - name: azure.resourcegroup


The azure.resource.group , created by a pipeline?

I'm not familiar with these integrations, but, yes, I see it being created by a grok in a few pipeline.

https://github.com/search?q=repo%3Aelastic%2Fintegrations+path%3A**%2F*.yml+%2Fazure%5C.resource%5C.%3Fgroup%2F&type=code

I understand that it then needs to be added in all the integrations because it comes from the processor.

This is a really general problem we have with Fleet integrations in that there is no easy or centralized way to account for the fields being added by the processors.

I hear your concerns and I will remove this field as it beats the purpose of this PR which was to add the orchestrator ecs fields.

👍

Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>

elasticmachine · 2024-03-11T11:05:39Z

💔 Build Failed

Buildkite Build
Commit: 785135f

Failed CI Steps

:rhel: RHEL-9 Unit Tests

History

💔 Build #1943 failed 665f9d5
💚 Build #1920 succeeded c74720c
💚 Build #1830 succeeded 757fc01
💚 Build #1799 succeeded d57fa5f
💚 Build #1791 succeeded 70865f3

cc @MichaelKatsoulis

elasticmachine · 2024-03-11T11:06:51Z

💔 Build Failed

Buildkite Build
Commit: 785135f

Failed CI Steps

History

💔 Build #3574 failed 665f9d5
💔 Build #3551 failed c74720c
💔 Build #3549 failed df6f6c5
💚 Build #3459 succeeded 757fc01

cc @MichaelKatsoulis

elasticmachine · 2024-03-11T11:07:07Z

💔 Build Failed

Buildkite Build
Commit: 785135f

Failed CI Steps

History

💔 Build #2233 failed 665f9d5
💔 Build #2210 failed c74720c
💔 Build #2208 failed df6f6c5
💚 Build #2119 succeeded 757fc01

cc @MichaelKatsoulis

elasticmachine · 2024-03-11T11:33:04Z

💚 Build Succeeded

Buildkite Build
Commit: 785135f

History

💚 Build #1936 succeeded 665f9d5
💚 Build #1913 succeeded c74720c
💚 Build #1911 succeeded df6f6c5
💚 Build #1823 succeeded 757fc01
💚 Build #1792 succeeded d57fa5f
💚 Build #1784 succeeded 70865f3

cc @MichaelKatsoulis

elasticmachine · 2024-03-11T11:38:03Z

💚 Build Succeeded

Buildkite Build
Commit: 785135f

History

💚 Build #1099 succeeded 665f9d5
💚 Build #1076 succeeded c74720c
💚 Build #986 succeeded 757fc01
💚 Build #955 succeeded d57fa5f
💚 Build #947 succeeded 70865f3
💚 Build #908 succeeded 0e08561

cc @MichaelKatsoulis

elasticmachine · 2024-03-11T11:46:39Z

💚 Build Succeeded

Buildkite Build
Commit: 785135f

History

💚 Build #3150 succeeded 665f9d5
💚 Build #3127 succeeded c74720c
💚 Build #3037 succeeded 757fc01
💚 Build #3006 succeeded d57fa5f
💚 Build #2998 succeeded 70865f3
💚 Build #2959 succeeded 0e08561

cc @MichaelKatsoulis

Retrieve aks cluster name and id

7e74460

MichaelKatsoulis requested review from a team as code owners January 22, 2024 12:02

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jan 22, 2024

MichaelKatsoulis marked this pull request as draft January 22, 2024 12:03

mergify bot assigned MichaelKatsoulis Jan 22, 2024

MichaelKatsoulis mentioned this pull request Jan 22, 2024

Investigate if it is possible to set orchestrator fields from Cloud provider kubernetes metadata #33081

Closed

MichaelKatsoulis added 2 commits January 24, 2024 16:41

Add tests

7906f04

Update docs

eada5b2

MichaelKatsoulis added Team:Cloud-Monitoring Label for the Cloud Monitoring team Team:Cloudnative-Monitoring Label for the Cloud Native Monitoring team labels Jan 24, 2024

botelastic bot removed needs_team Indicates that the issue/PR needs a Team:* label labels Jan 24, 2024

MichaelKatsoulis requested review from zmoog, gizas and tetianakravchenko January 24, 2024 15:25

MichaelKatsoulis added 3 commits January 25, 2024 12:06

Lint errors and notice txt

c8d065a

make check updates

2c1095d

Update tests

47b4fea

MichaelKatsoulis added 3 commits January 25, 2024 16:15

Merge remote-tracking branch 'upstream/main' into add-aks-cluster-nam…

d6da3c5

…e-id

Use fake credentials

775afc9

Unused logger

d778f99

MichaelKatsoulis added 2 commits March 5, 2024 10:02

Update changelog

70865f3

Add azure.resourcegroup.name field under cloud key in fields.yml

d57fa5f

fearful-symmetry reviewed Mar 5, 2024

View reviewed changes

aleksmaus approved these changes Mar 5, 2024

View reviewed changes

Add fmt.Errorf in error returns

757fc01

fearful-symmetry approved these changes Mar 6, 2024

View reviewed changes

MichaelKatsoulis requested a review from andrewkroh March 7, 2024 08:50

andrewkroh reviewed Mar 7, 2024

View reviewed changes

MichaelKatsoulis and others added 7 commits March 8, 2024 11:30

Remove azure.resourcegroup.name

884af33

Update libbeat/processors/add_cloud_metadata/docs/add_cloud_metadata.…

e8c5400

…asciidoc Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>

Update libbeat/processors/add_cloud_metadata/docs/add_cloud_metadata.…

2ce7d88

…asciidoc Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>

Update libbeat/processors/add_cloud_metadata/provider_azure_vm.go

17a1286

Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>

Update libbeat/processors/add_cloud_metadata/provider_azure_vm.go

99db284

Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>

Update libbeat/processors/add_cloud_metadata/provider_azure_vm.go

df6f6c5

Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>

Update Id to ID

c74720c

MichaelKatsoulis requested a review from andrewkroh March 8, 2024 10:01

andrewkroh approved these changes Mar 8, 2024

View reviewed changes

Update libbeat/processors/add_cloud_metadata/provider_azure_vm.go

665f9d5

Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>

MichaelKatsoulis enabled auto-merge (squash) March 11, 2024 10:55

MichaelKatsoulis disabled auto-merge March 11, 2024 10:55

Merge branch 'main' into add-aks-cluster-name-id

785135f

pierrehilbert approved these changes Mar 11, 2024

View reviewed changes

MichaelKatsoulis removed the request for review from faec March 11, 2024 14:12

MichaelKatsoulis merged commit 7987272 into elastic:main Mar 11, 2024
160 of 164 checks passed

add_cloud_metadata - Retrieve AKS cluster name and id #37685

add_cloud_metadata - Retrieve AKS cluster name and id #37685

Conversation

MichaelKatsoulis commented Jan 22, 2024 • edited Loading

Proposed commit message

Checklist

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

mergify bot commented Jan 22, 2024

elasticmachine commented Jan 22, 2024

💔 Build Failed

Build stats

Pipeline error

❕ Flaky test report

🤖 GitHub comments

elasticmachine commented Jan 24, 2024

❕ Build Aborted

Build stats

🤖 GitHub comments

elasticmachine commented Jan 24, 2024

💔 Tests Failed

Build stats

Test stats 🧪

Test errors

Build&Test / libbeat-unitTest / TestRetrieveAzureMetadata – github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata

Build&Test / libbeat-goIntegTest / TestRetrieveAzureMetadata – github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata

Steps errors

libbeat-unitTest - mage build unitTest

libbeat-unitTest - mage build unitTest

libbeat-unitTest - mage build unitTest

libbeat-goIntegTest - mage goIntegTest

libbeat-goIntegTest - mage goIntegTest

libbeat-goIntegTest - mage goIntegTest

Error signal

🐛 Flaky test report

Genuine test errors

🤖 GitHub comments

elasticmachine commented Jan 25, 2024

❕ Build Aborted

Build stats

Test stats 🧪

Steps errors

Archive the artifacts

Checks if running on a Unix-like node

Archive the artifacts

Checks if running on a Unix-like node

Error signal

🤖 GitHub comments

elasticmachine commented Jan 25, 2024

❕ Build Aborted

Build stats

Test stats 🧪

Test errors

Build&Test / libbeat-unitTest / TestRetrieveAzureMetadata – github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata

Build&Test / libbeat-goIntegTest / TestRetrieveAzureMetadata – github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata

Steps errors

libbeat-unitTest - mage build unitTest

libbeat-unitTest - mage build unitTest

libbeat-unitTest - mage build unitTest

libbeat-goIntegTest - mage goIntegTest

libbeat-goIntegTest - mage goIntegTest

libbeat-goIntegTest - mage goIntegTest

Error signal

🤖 GitHub comments

elasticmachine commented Jan 25, 2024

💔 Tests Failed

Build stats

Test stats 🧪

Test errors

Build&Test / libbeat-unitTest / TestRetrieveAzureMetadata – github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata

Build&Test / libbeat-goIntegTest / TestRetrieveAzureMetadata – github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata

Steps errors

libbeat-unitTest - mage build unitTest

libbeat-unitTest - mage build unitTest

libbeat-unitTest - mage build unitTest

libbeat-goIntegTest - mage goIntegTest

libbeat-goIntegTest - mage goIntegTest

MichaelKatsoulis commented Jan 22, 2024 •

edited

Loading

`Build&Test / libbeat-unitTest / TestRetrieveAzureMetadata – github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata`

`Build&Test / libbeat-goIntegTest / TestRetrieveAzureMetadata – github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata`

`libbeat-unitTest - mage build unitTest`

`libbeat-unitTest - mage build unitTest`

`libbeat-unitTest - mage build unitTest`

`libbeat-goIntegTest - mage goIntegTest`

`libbeat-goIntegTest - mage goIntegTest`

`libbeat-goIntegTest - mage goIntegTest`

`Error signal`

`Archive the artifacts`

`Checks if running on a Unix-like node`

`Archive the artifacts`

`Checks if running on a Unix-like node`

`Error signal`

`Build&Test / libbeat-unitTest / TestRetrieveAzureMetadata – github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata`

`Build&Test / libbeat-goIntegTest / TestRetrieveAzureMetadata – github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata`

`libbeat-unitTest - mage build unitTest`

`libbeat-unitTest - mage build unitTest`

`libbeat-unitTest - mage build unitTest`

`libbeat-goIntegTest - mage goIntegTest`

`libbeat-goIntegTest - mage goIntegTest`

`libbeat-goIntegTest - mage goIntegTest`

`Error signal`

`Build&Test / libbeat-unitTest / TestRetrieveAzureMetadata – github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata`

`Build&Test / libbeat-goIntegTest / TestRetrieveAzureMetadata – github.com/elastic/beats/v7/libbeat/processors/add_cloud_metadata`

`libbeat-unitTest - mage build unitTest`

`libbeat-unitTest - mage build unitTest`

`libbeat-unitTest - mage build unitTest`

`libbeat-goIntegTest - mage goIntegTest`

`libbeat-goIntegTest - mage goIntegTest`

`libbeat-goIntegTest - mage goIntegTest`

`Error signal`

elasticmachine commented Mar 11, 2024 •

edited

Loading

elasticmachine commented Mar 11, 2024 •

edited

Loading

elasticmachine commented Mar 11, 2024 •

edited

Loading