Implement Kubernetes Metadata Extension #1583

musa-asad · 2025-03-06T05:15:41Z

Description of the issue

To support the Explore related feature in CloudWatch, the CloudWatch Agent sends an "Entity", which includes relevant metadata to correlate metrics or logs between resources (e.g., an EKS cluster) and services (e.g., a Java application). When the CloudWatch Agent runs in a Kubernetes cluster, we need to collect the namespace, workload name, and node name to populate the "Entity".

However, we currently only get Kubernetes metadata when Application Signals is enabled. For OTLP custom metrics, if Application Signals isn't configured, then we don't have a way to fetch Kubernetes metadata. To achieve this, we must migrate the logic the Application Signals Processor uses to get Kuberbetes metadata to a global extension, which can be used by other pipelines.

Description of changes

Implement Kubernetes Metadata Extension (extension/k8smetadata)
- Add README.md to document how the extension works.
- Add config.go and factory.go barebone files in order to use in translation for the OTEL configuration.
- Add extension.go, which sets up an EndpointSlice watcher and Pod IP → {Workload, Namespace, Node} mappings. The GetPodMetadata() method returns the respective mapping for a given Pod IP.
Replicate endpointslicewatcher.go and kubernetes_utils.go to the internal/k8sCommon/k8sclient directory and adjust Pod IP → Workload@Namespace logic to Pod IP → {Workload, Namespace, Node}. Unused functionality was removed.
Update Entity Processor to Use Extension (awsentity)
- Implement getPodMeta() function in processor.go, which invokes the Kubernetes Metadata Extension to receive {Workload, Namespace, Node} from a given Pod IP.
- Update internal/k8sattributescraper/k8sattributescraper.go to use the Kubernetes Metadata Extension to populate workload, namespace, and node instead of resource attributes if those values are present.
Add Kubernetes Metadata Extension to Translation for the OTLP Pipeline (translator/translate/otel)
- Update sample yaml files to include new extension.
- Create extension/k8smetadata/translator.go to reference the extension in translation.
- Add translators.Extensions.Set(k8smetadata.NewTranslator()) to pipeline/host/translator.go for OTLP pipeline in Kubernetes contexts.
Update service/defaultcomponents/components.go to include "k8smetadata".
Add unit tests for new functionality.

License

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Tests

Created an EKS cluster and deployed the Amazon CloudWatch Observability EKS add-on.
Set up sample application by following https://aws-otel.github.io/docs/getting-started/adot-eks-add-on/sample-app.

Removed resource attributes.
Changed OTEL_EXPORTER_OTLP_ENDPOINT to http://cloudwatch-agent.amazon-cloudwatch:4317.

Built the agent image by running make docker-build-amd64 and changed the image in the AmazonCloudWatchAgent custom resource from k8s-metadata-extension branch.

Added debug exporter to OTLP pipeline for testing.

Agent Config:

    {
      "agent": {
       "debug": true,
      },
      "logs": {
        "metrics_collected": {
          "otlp": {
            "grpc_endpoint": "0.0.0.0:4317",
            "http_endpoint": "0.0.0.0:4318"
          },
        }
      },
    }

Kubernetes Metadata Extension

Logs:

2025-03-10T06:26:12Z I! {"caller":"k8smetadata/extension.go:70","msg":"Starting EndpointSliceWatcher Run()","kind":"extension","name":"k8smetadata"}                                                                                                                               
2025-03-10T06:26:12Z I! {"caller":"k8smetadata/extension.go:72","msg":"Waiting for EndpointSlice cache to sync...","kind":"extension","name":"k8smetadata"}                                                                                                                        
I0310 06:26:12.913113       1 shared_informer.go:313] Waiting for caches to sync for endpointSliceWatcher                                 I0310 06:26:13.013709       1 shared_informer.go:320] Caches are synced for endpointSliceWatcher                                          2025-03-10T06:26:13Z I! {"caller":"k8sclient/endpointslicewatcher.go:83","msg":"endpointSliceWatcher: Cache synced","kind":"extension","name":"k8smetadata"}
2025-03-10T06:26:13Z I! {"caller":"k8smetadata/extension.go:74","msg":"EndpointSlice cache synced, extension fully started","kind":"extension","name":"k8smetadata"}
2025-03-10T04:57:35Z D! {"caller":"k8sclient/endpointslicewatcher.go:111","msg":"Processing endpoint","kind":"extension","name":"k8smetadata","podName":"sample-app-786d6c49b4-j4vvx","namespace":"default","nodeName":"ip-XXX-XX-XX-XX.us-west-2.compute.internal"}
2025-03-10T04:57:35Z D! {"caller":"k8smetadata/extension.go:93","msg":"GetPodMetadata: found metadata","kind":"extension","name":"k8smetadata","ip":"XXX.XX.XX.XXX","workload":"sample-app","namespace":"default","node":"ip-XXX-XX-XX-XX.us-west-2.compute.internal"}

Debug Exporter Output:
Entity Fields:

com.amazonaws.cloudwatch.entity.internal.type: Str(Service)
com.amazonaws.cloudwatch.entity.internal.service.name: Str(sample-app)
com.amazonaws.cloudwatch.entity.internal.deployment.environment: Str(eks:entity-cluster-2/default)
com.amazonaws.cloudwatch.entity.internal.platform.type: Str(AWS::EKS)
com.amazonaws.cloudwatch.entity.internal.k8s.cluster.name: Str(entity-cluster-2)
com.amazonaws.cloudwatch.entity.internal.k8s.namespace.name: Str(default)
com.amazonaws.cloudwatch.entity.internal.k8s.workload.name: Str(sample-app)
com.amazonaws.cloudwatch.entity.internal.k8s.node.name: Str(ip-XXX-XX-XX-XX.us-west-2.compute.internal)
com.amazonaws.cloudwatch.entity.internal.instance.id: Str(i-006ac48ffed131779)

Requirements

Before commit the code, please do the following steps.

Run make fmt and make fmt-sh
Run make lint

Co-authored-by: Ping Xiang <>

extension/k8smetadata/extension.go

internal/k8sCommon/k8sclient/endpointslicewatcher.go

Co-authored-by: Lisa Guo <lguo25@gmail.com>

nathalapooja

OverAll looks good. Added few comments and check the below as well

It says it used 'k8s-attr` branch in overview instead of PR branch
Lets do stress test to make sure we are not throttling the k8s api.

nathalapooja · 2025-03-12T12:45:58Z

extension/k8smetadata/extension.go

+		return k8sclient.PodMetadata{}
+	}
+	metadata := pm.(k8sclient.PodMetadata)
+	e.logger.Debug("GetPodMetadata: found metadata",


Agent should not be logging entity key attributes and attributes.

We already log them as part of the EMF exporter so I assumed that was okay, but that's a good point. We should not log them in both cases.

extension/k8smetadata/extension.go

nathalapooja · 2025-03-12T12:49:51Z

extension/k8smetadata/extension_test.go

+	}
+
+	const testIP = "1.2.3.4"
+	expected := k8sclient.PodMetadata{


Is there a possibility that podMetadata has only workload/any one field and remaining as empty?

As in, namespace and node name is empty, but workload is present? It's possible, but I don't believe it would happen. I can add testing coverage for it though.

internal/k8sCommon/k8sclient/endpointslicewatcher.go

internal/k8sCommon/k8sclient/kubernetes_utils.go

nathalapooja · 2025-03-12T13:13:22Z

internal/k8sCommon/k8sclient/kubernetes_utils_test.go

+			service:  "nginx-service",
+			expected: "nginx-123456",
+		},
+		{


Write a test to fallback to fullpodname

nathalapooja · 2025-03-12T14:47:48Z

plugins/processors/awsentity/processor.go

@@ -176,7 +193,7 @@ func (p *awsEntityProcessor) processMetrics(_ context.Context, md pmetric.Metric
 				}
 			}
 			if p.config.KubernetesMode != "" {
-				p.k8sscraper.Scrape(rm.At(i).Resource())
+				p.k8sscraper.Scrape(rm.At(i).Resource(), getPodMeta())


Add additional unit tests for validating the addition of entity fields by reading from podMeta

This reverts commit ca56a5b.

musa-asad changed the base branch from main to feature-custom-metrics-entity March 6, 2025 05:16

musa-asad force-pushed the k8s-metadata-extension branch from a5ea567 to 512f3f3 Compare March 6, 2025 05:17

musa-asad closed this Mar 6, 2025

musa-asad force-pushed the k8s-metadata-extension branch from 512f3f3 to 20824f2 Compare March 6, 2025 05:20

musa-asad reopened this Mar 6, 2025

musa-asad self-assigned this Mar 6, 2025

Implement Kubernetes Metadata Extension

801cd88

Co-authored-by: Ping Xiang <>

musa-asad force-pushed the k8s-metadata-extension branch 3 times, most recently from e9a7edf to 801cd88 Compare March 10, 2025 06:02

Merge branch 'feature-custom-metrics-entity' into k8s-metadata-extension

8f433ff

musa-asad marked this pull request as ready for review March 10, 2025 06:30

musa-asad requested a review from a team as a code owner March 10, 2025 06:30

fix lint

4356143

musa-asad requested review from lisguo and nathalapooja March 10, 2025 06:35

Fix lint

57bfa36

lisguo reviewed Mar 11, 2025

View reviewed changes

musa-asad and others added 2 commits March 11, 2025 20:49

Fix comment.

9906d56

Co-authored-by: Lisa Guo <lguo25@gmail.com>

rename

ca56a5b

nathalapooja reviewed Mar 12, 2025

View reviewed changes

musa-asad added 2 commits March 12, 2025 12:09

Revert "rename"

b4f386a

This reverts commit ca56a5b.

add tests; fix race condition

e024ef6

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement Kubernetes Metadata Extension #1583

Implement Kubernetes Metadata Extension #1583

musa-asad commented Mar 6, 2025 •

edited

Loading

nathalapooja left a comment

nathalapooja Mar 12, 2025

musa-asad Mar 12, 2025

nathalapooja Mar 12, 2025

musa-asad Mar 12, 2025

nathalapooja Mar 12, 2025

nathalapooja Mar 12, 2025

Implement Kubernetes Metadata Extension #1583

Are you sure you want to change the base?

Implement Kubernetes Metadata Extension #1583

Conversation

musa-asad commented Mar 6, 2025 • edited Loading

Description of the issue

Description of changes

License

Tests

Kubernetes Metadata Extension

Requirements

nathalapooja left a comment

Choose a reason for hiding this comment

nathalapooja Mar 12, 2025

Choose a reason for hiding this comment

musa-asad Mar 12, 2025

Choose a reason for hiding this comment

nathalapooja Mar 12, 2025

Choose a reason for hiding this comment

musa-asad Mar 12, 2025

Choose a reason for hiding this comment

nathalapooja Mar 12, 2025

Choose a reason for hiding this comment

nathalapooja Mar 12, 2025

Choose a reason for hiding this comment

musa-asad commented Mar 6, 2025 •

edited

Loading