The parameter logs.metrics_collected.prometheus.emf_processor.metric_declaration.metric_selectors without any function

[eugen-eugen]

Describe the bug
I whant to use the parameter logs.metrics_collected.prometheus.emf_processor.metric_declaration.metric_selectors in order to send only metrics I whant to AWS Cloud Watch, as described in https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/ContainerInsights-Prometheus-Setup-configure.html:

_metric_declaration— are sections that specify the array of logs with embedded metric format to be generated. There are metric_declaration sections for each Prometheus source that the CloudWatch agent imports from by default. These sections each include the following fields:

label_matcher is a regular expression that checks the value of the labels listed in source_labels. The metrics that match are enabled for inclusion in the embedded metric format sent to CloudWatch.

If you have multiple labels specified in source_labels, we recommend that you do not use ^ or $ characters in the regular expression for label_matcher.

source_labels specifies the value of the labels that are checked by the label_matcher line.

label_separator specifies the separator to be used in the label_matcher line if multiple source_labels are specified. The default is ;. You can see this default used in the label_matcher line in the following example.

metric_selectors is a regular expression that specifies the metrics to be collected and sent to CloudWatch.

dimensions is the list of labels to be used as CloudWatch dimensions for each selected metric._

====================================================================================
But no matter how i set metric_selectors, it doesn't have any effect: all scrapped metrics are sent to Cloud Watch.

Steps to reproduce
Set metric_selectors to any existing metric name

What did you expect to see?
Only this metric in Cloud Watch

What did you see instead?
All metrics which prometheus had scrapped from my application

What version did you use?
1.300051.0b992

What config did you use?

....
"emf_processor": {
              "metric_declaration": [
                {
                  "source_labels": ["*"],
                  "label_matcher": "*spring*",
                  "dimensions": [["ClusterName","Namespace", "Endpoint"]],
                  "metric_selectors": [
                    "^spring_data_repository_invocations_seconds_max$"
                  ]
                }
              ]
            }
...

Environment
OS: Ubuntu

Additional context
I've researched in the code of cw agent here on github and i didn't find any usage of the variable MetricSeclector, which is set by the parameter metric_selctor.
My cloud watch agent runs inside of a EKS-Cluster in the same AWS-Account as Cloud Watch.

[jefchien]

The metric selectors get used in the EMF exporter as a way to filter the metrics and metrics that don't match any of the patterns should get dropped.

As an example,

amazon-cloudwatch-agent/translator/tocwconfig/sampleConfig/prometheus_config_linux.json

Lines 55 to 82 in 5e3b831

	"metric_declaration": [
	{
	"dimensions": [["Service"]],
	"label_matcher": "nginx.*",
	"label_separator": ";",
	"metric_selectors": ["^nginx_request_count$"],
	"source_labels": ["Service"]
	},
	{
	"label_matcher": "default",
	"metric_selectors": [".*"],
	"source_labels": ["Namespace"]
	},
	{
	"source_labels":["name"],
	"dimensions":[
	["name"]
	],
	"metric_selectors": ["^.*$"]
	},
	{
	"source_labels":["name"],
	"dimensions":[
	["name"]
	],
	"metric_selectors": ["^node_cpu_guest_seconds_total$"]
	}
	]

becomes

amazon-cloudwatch-agent/translator/tocwconfig/sampleConfig/prometheus_config_linux.yaml

Lines 16 to 55 in 5e3b831

	metric_declarations:
	- dimensions:
	- - Service
	label_matchers:
	- label_names:
	- Service
	regex: nginx.*
	separator: ;
	metric_name_selectors:
	- ^nginx_request_count$
	- dimensions: []
	label_matchers:
	- label_names:
	- Namespace
	regex: default
	separator: ;
	metric_name_selectors:
	- .*
	- dimensions:
	- - name
	label_matchers:
	- label_names:
	- name
	regex: .*
	separator: ;
	metric_name_selectors:
	- ^.*$
	- dimensions:
	- - name
	label_matchers:
	- label_names:
	- name
	regex: .*
	separator: ;
	metric_name_selectors:
	- ^node_cpu_guest_seconds_total$
	metric_descriptors:
	- metric_name: nginx_request_count
	overwrite: false
	unit: Count

Can you provide a more complete configuration?

[eugen-eugen]

Thank you for the answer. Yes, of course, here is it:

apiVersion: v1
data:
  # cwagent json config
  # debug aktiviert
  cwagentconfig.json: |
    {
      "agent": {
        "debug": true
      },
      "logs": {
        "metrics_collected": {
          "prometheus": {
            "log_group_name": "ymo-promethes",
            "prometheus_config_path": "/etc/prometheusconfig/prometheus.yaml",
            "emf_processor": {
              "metric_declaration": [
                {
                  "source_labels": ["job"],
                  "label_matcher": "*spring*",
                  "dimensions": [["ClusterName","Namespace", "Endpoint"]],
                  "metric_selectors": [
                    "^spring_data_repository_invocations_seconds_max$"
                  ]
                }
              ]
            }
          }
        },
        "force_flush_interval": 5
      }
    }    
kind: ConfigMap
metadata:
  name: ymo-prometheus-cwagentconfig
  namespace: default
---
apiVersion: v1
data:
  # prometheus config
  prometheus.yaml: |
    global:
      scrape_interval: 1m
      scrape_timeout: 10s
    scrape_configs:
    - job_name: 'ymo-sd-kubernetes'
      sample_limit: 10000
      metrics_path: /actuator/prometheus
      kubernetes_sd_configs:
        - role: endpoints
      scheme: http
      relabel_configs:
        - source_labels: [__meta_kubernetes_namespace]
          action: replace
          target_label: Namespace      
        - source_labels: [__meta_kubernetes_endpoints_name]
          action: replace
          target_label: Endpoint      
          
kind: ConfigMap
metadata:
  name: ymo-prometheus-config
  namespace: default
---
# Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ymo-cwagent-prometheus
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ymo-cwagent-prometheus
  template:
    metadata:
      labels:
        app: ymo-cwagent-prometheus
    spec:
      containers:
        - name: ymo-cloudwatch-agent

          image: public.ecr.aws/cloudwatch-agent/cloudwatch-agent
          imagePullPolicy: Always
          resources:
            limits:
              cpu:  1000m
              memory: 1000Mi
            requests:
              cpu: 200m
              memory: 200Mi
          # Please don't change below envs
          env:
            - name: CI_VERSION
              value: "k8s/1.3.29"
          # Please don't change the mountPath
          volumeMounts:
            - name: ymo-prometheus-cwagentconfig
              mountPath: /etc/cwagentconfig
            - name: ymo-prometheus-config
              mountPath: /etc/prometheusconfig

      volumes:
        - name: ymo-prometheus-cwagentconfig
          configMap:
            name: ymo-prometheus-cwagentconfig
        - name: ymo-prometheus-config
          configMap:
            name: ymo-prometheus-config
      terminationGracePeriodSeconds: 60
      serviceAccountName: cwagent-prometheus

The metric spring_data_repository_invocations_seconds_max is only one I want to have in cloud watch. But I see all of them of my application.
BTW: my cloud watch agent runs inside of a EKS-Cluster in the same AWS-Account as Cloud Watch

[github-actions]

This issue was marked stale due to lack of activity.

[sky333999]

Could you also please provide us with a sanitized snapshot of your raw prometheus metrics so we can try to reproduce the issue?