Relabelling the address of a TargetGroup from k8s namespace, results in duplication of metrics

[ionutvilie]

What did you do?

replace the address label using a group regex where port is optional from data received from kubernetes_sd_configs

related to this: #2434

What did you expect to see?
one metric with the label address changed

What did you see instead? Under which circumstances?

i am seeing the targets reported correctly, the metric however depending on how many regex groups i use it is doubled or tripled

Environment

• System information:

Linux 3.13.0-85-generic x86_64

• Prometheus version:

prometheus, version 1.5.2 (branch: master, revision: bd1182d29f462c39544f94cc822830e1c64cf55b)
  build user:       root@a8af9200f95d
  build date:       20170210-14:41:22
  go version:       go1.7.5

I have checked new versions release notes and nothing seems to be related to any regex fix.
From my point of view is not relevant to update to the newer version.

• Prometheus configuration file:

############################################################
# Configuration for K8S cluster 'mcc1-test'
############################################################
  ################################################
  # Configuration for foo services
  ################################################
  - job_name: 'foo-mcc1-test'

    scrape_interval: 15s
    metrics_path: /.well-known/metrics

    proxy_url: 'https://proxy-k8s-001-test1-mcc-gb-lon1.metroscales.io:30040'

    kubernetes_sd_configs:
    - role: pod
      api_server: 'api-foo.io'
      basic_auth:
        username: foo
        password: bar

    relabel_configs:
    # Drop target which should not be monitored
    - source_labels: [__meta_kubernetes_namespace]
      action: drop
      regex: default|kube-system|foo-monitoring
      target_label: __meta_kubernetes_namespace
    # Add foo_labels
    - source_labels: [__meta_kubernetes_pod_label_vertical]
      action: replace
      regex: (.+)
      replacement: ${1}
      target_label: foo_vertical
    - source_labels: [__meta_kubernetes_pod_label_stage]
      action: replace
      regex: (.+)
      replacement: ${1}
      target_label: foo_stage
    - source_labels: [__meta_kubernetes_pod_label_service]
      action: replace
      regex: (.+)
      replacement: ${1}
      target_label: foo_service
    ## below one triples the metric (one IP) 3 hostnames  
    # - source_labels: [__address__,__meta_kubernetes_pod_host_ip]
    #   separator: ;
    #   action: replace
    #   regex: ([\.0-9]+)(?:(:[0-9]+)?);([\.0-9]+)
    #   replacement: ${3}${2}
    #   target_label: __address__
    ## below one was inserted to trace the bug  
    # - source_labels: [__meta_kubernetes_pod_name]
    #   action: replace
    #   regex: (.+)
    #   target_label: testxxx
    ## below one doubles the metric (one IP) 2 hostnames  
    # - source_labels: [__meta_kubernetes_pod_host_ip, __meta_kubernetes_pod_container_port_number]
    #   action: replace
    #   target_label: __address__
    #   regex: (.*);(.*)
    #   replacement: $1:$2

• Screen Shot:

insert Prometheus and Alertmanager logs relevant to the issue here

[ionutvilie]

After hours of browsing the source code repository :), i have learned how to activate the debug and ended up at a structure definition. i was using the latest Prometheus release on my local machine.

My presumption is that when i am trying to relabel the __address__ label, the group looses it's unique ID and therefore i am getting the data multiplied by the number of regex group labels.

Is this correct and can someone at least confirm this ? i can reproduce it on multiple kubernetes clusters and i think it is can only be reproduce inside a k8s namespace.

also for the below structure wouldn't be a better idea to use __meta_kubernetes_pod_label_pod_template_hash ?

// TargetGroup is a set of targets with a common label set.
type TargetGroup struct {
	// Targets is a list of targets identified by a label set. Each target is
	// uniquely identifiable in the group by its address label.
	Targets []model.LabelSet
	// Labels is a set of labels that is common across all targets in the group.
	Labels model.LabelSet

	// Source is an identifier that describes a group of targets.
	Source string
}

func (tg TargetGroup) String() string {
	return tg.Source
}

• Logs:

time="2017-05-25T14:22:23+03:00" level=debug msg="initial pod" kubernetes_sd=pod source="pod.go:56" tg="&config.TargetGroup{Targets:[]model.LabelSet{model.LabelSet{"__meta_kubernetes_pod_container_name":"identity-idamweb",
"__meta_kubernetes_pod_container_port_number":"30223",
"__meta_kubernetes_pod_container_port_name":"",
"__meta_kubernetes_pod_container_port_protocol":"TCP",
"__address__":"172.20.91.4:30223"}},
Labels:model.LabelSet{"__meta_kubernetes_pod_node_name":"minion2-k8s3",
"__meta_kubernetes_pod_host_ip":"10.31.235.15",
"__meta_kubernetes_pod_label_service":"idamweb",
"__meta_kubernetes_pod_label_stage":"dev",
"__meta_kubernetes_pod_label_app":"identity-idamweb",
"__meta_kubernetes_pod_annotation_kubernetes_io_created_by":"{"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"identity-dev","name":"identity-idamweb-1534902401","uid":"5bbd1335-3eea-11e7-90e3-bc764e081ae9","apiVersion":"extensions","resourceVersion":"18313706"}}n",
"__meta_kubernetes_pod_ip":"172.20.91.4",
"__meta_kubernetes_pod_ready":"true",
"__meta_kubernetes_pod_label_pod_template_hash":"1534902401",
"__meta_kubernetes_pod_label_vertical":"identity",
"__meta_kubernetes_namespace":"identity-dev",
"__meta_kubernetes_pod_name":"identity-idamweb-1534902401-8nkwc"},
Source:"pod/identity-dev/identity-idamweb-1534902401-8nkwc"}" 

time="2017-05-25T14:22:23+03:00" level=debug msg="initial pod" kubernetes_sd=pod source="pod.go:56" tg="&config.TargetGroup{Targets:[]model.LabelSet{model.LabelSet{"__meta_kubernetes_pod_container_port_number":"30223",
"__meta_kubernetes_pod_container_port_name":"",
"__meta_kubernetes_pod_container_port_protocol":"TCP",
"__address__":"172.20.18.18:30223",
"__meta_kubernetes_pod_container_name":"identity-idamweb"}},
Labels:model.LabelSet{"__meta_kubernetes_pod_label_vertical":"identity",
"__meta_kubernetes_pod_annotation_kubernetes_io_created_by":"{"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"identity-dev","name":"identity-idamweb-1534902401","uid":"5bbd1335-3eea-11e7-90e3-bc764e081ae9","apiVersion":"extensions","resourceVersion":"18313911"}}n",
"__meta_kubernetes_namespace":"identity-dev",
"__meta_kubernetes_pod_host_ip":"10.31.235.8",
"__meta_kubernetes_pod_label_app":"identity-idamweb",
"__meta_kubernetes_pod_ready":"true",
"__meta_kubernetes_pod_node_name":"minion0-k8s3",
"__meta_kubernetes_pod_label_pod_template_hash":"1534902401",
"__meta_kubernetes_pod_label_service":"idamweb",
"__meta_kubernetes_pod_label_stage":"dev",
"__meta_kubernetes_pod_name":"identity-idamweb-1534902401-8g0jt",
"__meta_kubernetes_pod_ip":"172.20.18.18"},
Source:"pod/identity-dev/identity-idamweb-1534902401-8g0jt"}" 

time="2017-05-25T14:22:23+03:00" level=debug msg="initial pod" kubernetes_sd=pod source="pod.go:56" tg="&config.TargetGroup{Targets:[]model.LabelSet{model.LabelSet{"__meta_kubernetes_pod_container_name":"identity-idamweb",
"__meta_kubernetes_pod_container_port_number":"30223",
"__meta_kubernetes_pod_container_port_name":"",
"__meta_kubernetes_pod_container_port_protocol":"TCP",
"__address__":"172.20.37.11:30223"}},
Labels:model.LabelSet{"__meta_kubernetes_pod_ip":"172.20.37.11",
"__meta_kubernetes_pod_ready":"true",
"__meta_kubernetes_pod_node_name":"minion1-k8s3",
"__meta_kubernetes_pod_host_ip":"10.31.235.3",
"__meta_kubernetes_pod_label_service":"idamweb",
"__meta_kubernetes_pod_label_app":"identity-idamweb",
"__meta_kubernetes_pod_label_pod_template_hash":"1534902401",
"__meta_kubernetes_pod_name":"identity-idamweb-1534902401-wa64n",
"__meta_kubernetes_pod_label_vertical":"identity",
"__meta_kubernetes_pod_annotation_kubernetes_io_created_by":"{"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"identity-dev","name":"identity-idamweb-1534902401","uid":"5bbd1335-3eea-11e7-90e3-bc764e081ae9","apiVersion":"extensions","resourceVersion":"18313689"}}n",
"__meta_kubernetes_namespace":"identity-dev",
"__meta_kubernetes_pod_label_stage":"dev"},
Source:"pod/identity-dev/identity-idamweb-1534902401-wa64n"}" 


****


time="2017-05-25T14:22:23+03:00" level=debug msg="pod update" kubernetes_sd=pod source="pod.go:66" tg="&config.TargetGroup{Targets:[]model.LabelSet{model.LabelSet{"__address__":"172.20.37.11:30223",
"__meta_kubernetes_pod_container_name":"identity-idamweb",
"__meta_kubernetes_pod_container_port_number":"30223",
"__meta_kubernetes_pod_container_port_name":"",
"__meta_kubernetes_pod_container_port_protocol":"TCP"}},
Labels:model.LabelSet{"__meta_kubernetes_pod_name":"identity-idamweb-1534902401-wa64n",
"__meta_kubernetes_pod_ready":"true",
"__meta_kubernetes_pod_host_ip":"10.31.235.3",
"__meta_kubernetes_pod_label_stage":"dev",
"__meta_kubernetes_namespace":"identity-dev",
"__meta_kubernetes_pod_label_vertical":"identity",
"__meta_kubernetes_pod_annotation_kubernetes_io_created_by":"{"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"identity-dev","name":"identity-idamweb-1534902401","uid":"5bbd1335-3eea-11e7-90e3-bc764e081ae9","apiVersion":"extensions","resourceVersion":"18313689"}}n",
"__meta_kubernetes_pod_ip":"172.20.37.11",
"__meta_kubernetes_pod_node_name":"minion1-k8s3",
"__meta_kubernetes_pod_label_app":"identity-idamweb",
"__meta_kubernetes_pod_label_pod_template_hash":"1534902401",
"__meta_kubernetes_pod_label_service":"idamweb"},
Source:"pod/identity-dev/identity-idamweb-1534902401-wa64n"}" 

time="2017-05-25T14:22:23+03:00" level=debug msg="pod update" kubernetes_sd=pod source="pod.go:66" tg="&config.TargetGroup{Targets:[]model.LabelSet{model.LabelSet{"__address__":"172.20.91.4:30223",
"__meta_kubernetes_pod_container_name":"identity-idamweb",
"__meta_kubernetes_pod_container_port_number":"30223",
"__meta_kubernetes_pod_container_port_name":"",
"__meta_kubernetes_pod_container_port_protocol":"TCP"}},
Labels:model.LabelSet{"__meta_kubernetes_pod_label_pod_template_hash":"1534902401",
"__meta_kubernetes_pod_label_app":"identity-idamweb",
"__meta_kubernetes_namespace":"identity-dev",
"__meta_kubernetes_pod_name":"identity-idamweb-1534902401-8nkwc",
"__meta_kubernetes_pod_ip":"172.20.91.4",
"__meta_kubernetes_pod_host_ip":"10.31.235.15",
"__meta_kubernetes_pod_label_service":"idamweb",
"__meta_kubernetes_pod_label_stage":"dev",
"__meta_kubernetes_pod_label_vertical":"identity",
"__meta_kubernetes_pod_annotation_kubernetes_io_created_by":"{"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"identity-dev","name":"identity-idamweb-1534902401","uid":"5bbd1335-3eea-11e7-90e3-bc764e081ae9","apiVersion":"extensions","resourceVersion":"18313706"}}n",
"__meta_kubernetes_pod_ready":"true",
"__meta_kubernetes_pod_node_name":"minion2-k8s3"},
Source:"pod/identity-dev/identity-idamweb-1534902401-8nkwc"}" 

time="2017-05-25T14:22:23+03:00" level=debug msg="pod update" kubernetes_sd=pod source="pod.go:66" tg="&config.TargetGroup{Targets:[]model.LabelSet{model.LabelSet{"__address__":"172.20.18.18:30223",
"__meta_kubernetes_pod_container_name":"identity-idamweb",
"__meta_kubernetes_pod_container_port_number":"30223",
"__meta_kubernetes_pod_container_port_name":"",
"__meta_kubernetes_pod_container_port_protocol":"TCP"}},
Labels:model.LabelSet{"__meta_kubernetes_pod_name":"identity-idamweb-1534902401-8g0jt",
"__meta_kubernetes_pod_ready":"true",
"__meta_kubernetes_pod_host_ip":"10.31.235.8",
"__meta_kubernetes_pod_label_app":"identity-idamweb",
"__meta_kubernetes_pod_label_pod_template_hash":"1534902401",
"__meta_kubernetes_pod_label_stage":"dev",
"__meta_kubernetes_pod_label_vertical":"identity",
"__meta_kubernetes_pod_ip":"172.20.18.18",
"__meta_kubernetes_pod_node_name":"minion0-k8s3",
"__meta_kubernetes_pod_label_service":"idamweb",
"__meta_kubernetes_pod_annotation_kubernetes_io_created_by":"{"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"identity-dev","name":"identity-idamweb-1534902401","uid":"5bbd1335-3eea-11e7-90e3-bc764e081ae9","apiVersion":"extensions","resourceVersion":"18313911"}}n",
"__meta_kubernetes_namespace":"identity-dev"},
Source:"pod/identity-dev/identity-idamweb-1534902401-8g0jt"}" 


***


time="2017-05-25T14:22:24+03:00" level=debug msg="endpoints update" kubernetes_sd=endpoint source="endpoints.go:77" tg="&config.TargetGroup{Targets:[]model.LabelSet{model.LabelSet{"__meta_kubernetes_endpoint_port_protocol":"TCP",
"__meta_kubernetes_pod_container_name":"identity-idamweb",
"__meta_kubernetes_pod_container_port_protocol":"TCP",
"__meta_kubernetes_pod_label_stage":"dev",
"__meta_kubernetes_pod_container_port_name":"",
"__meta_kubernetes_pod_label_vertical":"identity",
"__meta_kubernetes_pod_ip":"172.20.18.18",
"__meta_kubernetes_pod_ready":"true",
"__meta_kubernetes_pod_label_service":"idamweb",
"__meta_kubernetes_pod_node_name":"minion0-k8s3",
"__meta_kubernetes_endpoint_port_name":"web",
"__meta_kubernetes_pod_annotation_kubernetes_io_created_by":"{"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"identity-dev","name":"identity-idamweb-1534902401","uid":"5bbd1335-3eea-11e7-90e3-bc764e081ae9","apiVersion":"extensions","resourceVersion":"18313911"}}n",
"__meta_kubernetes_pod_name":"identity-idamweb-1534902401-8g0jt",
"__meta_kubernetes_pod_container_port_number":"30223",
"__meta_kubernetes_endpoint_ready":"true",
"__address__":"172.20.18.18:30223",
"__meta_kubernetes_pod_host_ip":"10.31.235.8",
"__meta_kubernetes_pod_label_app":"identity-idamweb",
"__meta_kubernetes_pod_label_pod_template_hash":"1534902401"},
model.LabelSet{"__meta_kubernetes_pod_container_port_name":"",
"__meta_kubernetes_pod_name":"identity-idamweb-1534902401-wa64n",
"__meta_kubernetes_pod_label_stage":"dev",
"__meta_kubernetes_pod_annotation_kubernetes_io_created_by":"{"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"identity-dev","name":"identity-idamweb-1534902401","uid":"5bbd1335-3eea-11e7-90e3-bc764e081ae9","apiVersion":"extensions","resourceVersion":"18313689"}}n",
"__meta_kubernetes_pod_container_name":"identity-idamweb",
"__meta_kubernetes_pod_ready":"true",
"__meta_kubernetes_endpoint_port_name":"web",
"__meta_kubernetes_endpoint_ready":"true",
"__meta_kubernetes_pod_label_vertical":"identity",
"__meta_kubernetes_pod_node_name":"minion1-k8s3",
"__meta_kubernetes_pod_label_app":"identity-idamweb",
"__meta_kubernetes_pod_label_service":"idamweb",
"__meta_kubernetes_pod_container_port_number":"30223",
"__address__":"172.20.37.11:30223",
"__meta_kubernetes_pod_label_pod_template_hash":"1534902401",
"__meta_kubernetes_pod_host_ip":"10.31.235.3",
"__meta_kubernetes_pod_ip":"172.20.37.11",
"__meta_kubernetes_pod_container_port_protocol":"TCP",
"__meta_kubernetes_endpoint_port_protocol":"TCP"},
model.LabelSet{"__meta_kubernetes_pod_host_ip":"10.31.235.15",
"__meta_kubernetes_pod_annotation_kubernetes_io_created_by":"{"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"identity-dev","name":"identity-idamweb-1534902401","uid":"5bbd1335-3eea-11e7-90e3-bc764e081ae9","apiVersion":"extensions","resourceVersion":"18313706"}}n",
"__meta_kubernetes_pod_name":"identity-idamweb-1534902401-8nkwc",
"__meta_kubernetes_pod_label_vertical":"identity",
"__meta_kubernetes_pod_container_name":"identity-idamweb",
"__meta_kubernetes_pod_container_port_number":"30223",
"__meta_kubernetes_endpoint_port_protocol":"TCP",
"__meta_kubernetes_endpoint_ready":"true",
"__meta_kubernetes_pod_container_port_protocol":"TCP",
"__meta_kubernetes_pod_label_pod_template_hash":"1534902401",
"__meta_kubernetes_pod_label_service":"idamweb",
"__meta_kubernetes_pod_node_name":"minion2-k8s3",
"__meta_kubernetes_pod_label_app":"identity-idamweb",
"__meta_kubernetes_pod_ready":"true",
"__meta_kubernetes_pod_label_stage":"dev",
"__address__":"172.20.91.4:30223",
"__meta_kubernetes_endpoint_port_name":"web",
"__meta_kubernetes_pod_ip":"172.20.91.4",
"__meta_kubernetes_pod_container_port_name":""}},
Labels:model.LabelSet{"__meta_kubernetes_service_label_app":"identity-idamweb",
"__meta_kubernetes_namespace":"identity-dev",
"__meta_kubernetes_endpoints_name":"identity-idamweb",
"__meta_kubernetes_service_name":"identity-idamweb"},
Source:"endpoints/identity-dev/identity-idamweb"}" 

[ionutvilie]

more info after i have cloned the master repository and started experimenting:
it seems that it is memory leaking somewhere after the relabel it is done.
Fake targets appear progressively: first scrape 1 target, second scrape 2 targets, third scrape 3 targets and it remains as so.

[ionutvilie]

miss-configuration, the metrics are scraped correctly, the issue comes from the k8s loadbalancer.

[juliusv]

@ionutvilie Ah, was starting to look into this yesterday and didn't get anywhere yet - glad it got resolved :)

[ionutvilie]

@juliusv, i didn't have the time to answer but thanks for looking into it and sorry for the lost time. :)

Maybe In the feature it would be great if someone, from the community, gave some hints for were to look or check, code test. I didn't actually expect help but i was feeling very lonely. A few more days and was ready to start logging with "day 7, if someone ever find this i'm lost in the queue manager"

I probably now, know half of the Prometheus code and how to properly code a Golang app ;)

[juliusv]

@ionutvilie Heh, well at least you learned a lot of code now :)

Sorry, but we get a lot of filed issues, mailing list questions, IRC questions, etc., and with a lot of the time spent on Prometheus being "free" personal time, it's not always feasible to get to everything. Everyone's trying their best though.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.