An error has occurred during metrics gathering

[siarhei-makarevich]

Output of hostname:9280/metrics after big delay (~25 sec)

An error has occurred during metrics gathering:

12 error(s) occurred:

• collected metric servicegroup_state label:<name:"member" value:"stageapp01" > label:<name:"ns_instance" value:"netscalervpx1a" > label:<name:"servicegroup" value:"test-8116" > gauge:<value:0 > was collected before with the same name and label values
• collected metric servicegroup_average_time_to_first_byte label:<name:"member" value:"stageapp01" > label:<name:"ns_instance" value:"netscalervpx1a" > label:<name:"servicegroup" value:"test-8116" > gauge:<value:0 > was collected before with the same name and label values
• collected metric servicegroup_total_requests label:<name:"member" value:"stageapp01" > label:<name:"ns_instance" value:"netscalervpx1a" > label:<name:"servicegroup" value:"test-8116" > counter:<value:0 > was collected before with the same name and label values
• collected metric servicegroup_total_responses label:<name:"member" value:"stageapp01" > label:<name:"ns_instance" value:"netscalervpx1a" > label:<name:"servicegroup" value:"test-8116" > counter:<value:0 > was collected before with the same name and label values
• collected metric servicegroup_total_request_bytes label:<name:"member" value:"stageapp01" > label:<name:"ns_instance" value:"netscalervpx1a" > label:<name:"servicegroup" value:"test-8116" > counter:<value:0 > was collected before with the same name and label values
• collected metric servicegroup_total_response_bytes label:<name:"member" value:"stageapp01" > label:<name:"ns_instance" value:"netscalervpx1a" > label:<name:"servicegroup" value:"test-8116" > counter:<value:0 > was collected before with the same name and label values
• collected metric servicegroup_current_client_connections label:<name:"member" value:"stageapp01" > label:<name:"ns_instance" value:"netscalervpx1a" > label:<name:"servicegroup" value:"test-8116" > gauge:<value:0 > was collected before with the same name and label values
• collected metric servicegroup_surge_count label:<name:"member" value:"stageapp01" > label:<name:"ns_instance" value:"netscalervpx1a" > label:<name:"servicegroup" value:"test-8116" > gauge:<value:0 > was collected before with the same name and label values
• collected metric servicegroup_current_server_connections label:<name:"member" value:"stageapp01" > label:<name:"ns_instance" value:"netscalervpx1a" > label:<name:"servicegroup" value:"test-8116" > gauge:<value:0 > was collected before with the same name and label values
• collected metric servicegroup_server_established_connections label:<name:"member" value:"stageapp01" > label:<name:"ns_instance" value:"netscalervpx1a" > label:<name:"servicegroup" value:"test-8116" > gauge:<value:0 > was collected before with the same name and label values
• collected metric servicegroup_current_reuse_pool label:<name:"member" value:"stageapp01" > label:<name:"ns_instance" value:"netscalervpx1a" > label:<name:"servicegroup" value:"test-8116" > gauge:<value:0 > was collected before with the same name and label values
• collected metric servicegroup_max_clients label:<name:"member" value:"stageapp01" > label:<name:"ns_instance" value:"netscalervpx1a" > label:<name:"servicegroup" value:"test-8116" > gauge:<value:0 > was collected before with the same name and label values

[siarhei-makarevich]

Looks like exporter didn't expect such configuration on netscaler

add serviceGroup test-8116 HTTP -maxClient 0 -maxReq 0 -cip ENABLED Client-IP -usip NO -useproxyport YES -cltTimeout 180 -svrTimeout 360 -CKA YES -TCPB NO -CMP YES
bind lb vserver ssotest:ssl test-8116
bind serviceGroup test-8116 stageapp01 8116
bind serviceGroup test-8116 stageapp01 8160 -CustomServerID ""None""

when serviceGroup contains the same server but with different ports

[rokett]

hmmmm....I thought we had some service groups with that kind of configuration and I haven't seen this before. Maybe I'm wrong though. It will need some testing; hopefully I'll have some time to take a look at it next week if the workload allows.

[siarhei-makarevich]

thank you will it wold be great if you find time for that

[rokett]

It's been a while, a long while, and we still don't have this issue, but that is likely because we don;t have this same configuration. Pretty sure it can be fixed by just adding the port as a label on the metrics though.

Are you able to build the application yourself? If so I'll make some changes in a dev branch for you to pull down and test.

[jkomara]

I am also having this same issue. I do not have access to the devices to change the server name. I can figure out how to build the application if you are able to push those changes to a dev branch.

[vintury]

@rokett I try you dev branch and have same errors:

An error has occurred during metrics gathering:

60 error(s) occurred:
* collected metric servicegroup_state label:<name:"member" value:"10.1" > label:<name:"ns_instance" value:"10.1.0.40" > label:<name:"servicegroup" value:"perf-prom-prometheus-operator-alertmanager_monitoring_443_perf-prom-prometheus-operator-alertmanager_monitoring_9093_svc" > gauge:<value:0 >  was collected before with the same name and label values
* collected metric servicegroup_average_time_to_first_byte label:<name:"member" value:"10.1" > label:<name:"ns_instance" value:"10.1.0.40" > label:<name:"servicegroup" value:"perf-prom-prometheus-operator-alertmanager_monitoring_443_perf-prom-prometheus-operator-alertmanager_monitoring_9093_svc" > gauge:<value:0 >  was collected before with the same name and label values
* collected metric servicegroup_total_requests label:<name:"member" value:"10.1" > label:<name:"ns_instance" value:"10.1.0.40" > label:<name:"servicegroup" value:"perf-prom-prometheus-operator-alertmanager_monitoring_443_perf-prom-prometheus-operator-alertmanager_monitoring_9093_svc" > counter:<value:0 >  was collected before with the same name and label values
* collected metric servicegroup_total_responses label:<name:"member" value:"10.1" > label:<name:"ns_instance" value:"10.1.0.40" > label:<name:"servicegroup" value:"perf-prom-prometheus-operator-alertmanager_monitoring_443_perf-prom-prometheus-operator-alertmanager_monitoring_9093_svc" > counter:<value:0 >  was collected before with the same name and label values
* collected metric servicegroup_total_request_bytes label:<name:"member" value:"10.1" > label:<name:"ns_instance" value:"10.1.0.40" > label:<name:"servicegroup" value:"perf-prom-prometheus-operator-alertmanager_monitoring_443_perf-prom-prometheus-operator-alertmanager_monitoring_9093_svc" > counter:<value:0 >  was collected before with the same name and label values
* collected metric servicegroup_total_response_bytes label:<name:"member" value:"10.1" > label:<name:"ns_instance" value:"10.1.0.40" > label:<name:"servicegroup" value:"perf-prom-prometheus-operator-alertmanager_monitoring_443_perf-prom-prometheus-operator-alertmanager_monitoring_9093_svc" > counter:<value:0 >  was collected before with the same name and label values
* collected metric servicegroup_current_client_connections label:<name:"member" value:"10.1" > label:<name:"ns_instance" value:"10.1.0.40" > label:<name:"servicegroup" value:"perf-prom-prometheus-operator-alertmanager_monitoring_443_perf-prom-prometheus-operator-alertmanager_monitoring_9093_svc" > gauge:<value:0 >  was collected before with the same name and label values
* collected metric servicegroup_surge_count label:<name:"member" value:"10.1" > label:<name:"ns_instance" value:"10.1.0.40" > label:<name:"servicegroup" value:"perf-prom-prometheus-operator-alertmanager_monitoring_443_perf-prom-prometheus-operator-alertmanager_monitoring_9093_svc" > gauge:<value:0 >  was collected before with the same name and label values
* collected metric servicegroup_current_server_connections label:<name:"member" value:"10.1" > label:<name:"ns_instance" value:"10.1.0.40" > label:<name:"servicegroup" value:"perf-prom-prometheus-operator-alertmanager_monitoring_443_perf-prom-prometheus-operator-alertmanager_monitoring_9093_svc" > gauge:<value:0 >  was collected before with the same name and label values
* collected metric servicegroup_server_established_connections label:<name:"member" value:"10.1" > label:<name:"ns_instance" value:"10.1.0.40" > label:<name:"servicegroup" value:"perf-prom-prometheus-operator-alertmanager_monitoring_443_perf-prom-prometheus-operator-alertmanager_monitoring_9093_svc" > gauge:<value:0 >  was collected before with the same name and label values
* collected metric servicegroup_current_reuse_pool label:<name:"member" value:"10.1" > label:<name:"ns_instance" value:"10.1.0.40" > label:<name:"servicegroup" value:"perf-prom-prometheus-operator-alertmanager_monitoring_443_perf-prom-prometheus-operator-alertmanager_monitoring_9093_svc" > gauge:<value:0 >  was collected before with the same name and label values
* collected metric servicegroup_max_clients label:<name:"member" value:"10.1" > label:<name:"ns_instance" value:"10.1.0.40" > label:<name:"servicegroup" value:"perf-prom-prometheus-operator-alertmanager_monitoring_443_perf-prom-prometheus-operator-alertmanager_monitoring_9093_svc" > gauge:<value:0 >  was collected before with the same name and label values
* collected metric servicegroup_state label:<name:"member" value:"10.1" > label:<name:"ns_instance" value:"10.1.0.40" > label:<name:"servicegroup" value:"perf-prom-prometheus-operator-alertmanager_monitoring_443_perf-prom-prometheus-operator-alertmanager_monitoring_9093_svc" > gauge:<value:0 >  was collected before with the same name and label values
* collected metric servicegroup_average_time_to_first_byte label:<name:"member" value:"10.1" > label:<name:"ns_instance" value:"10.1.0.40" > label:<name:"servicegroup" value:"perf-prom-prometheus-operator-alertmanager_monitoring_443_perf-prom-prometheus-operator-alertmanager_monitoring_9093_svc" > gauge:<value:0 >  was collected before with the same name and label values
* collected metric servicegroup_total_requests label:<name:"member" value:"10.1" > label:<name:"ns_instance" value:"10.1.0.40" > label:<name:"servicegroup" value:"perf-prom-prometheus-operator-alertmanager_monitoring_443_perf-prom-prometheus-operator-alertmanager_monitoring_9093_svc" > counter:<value:0 >  was collected before with the same name and label values
* collected metric servicegroup_total_responses label:<name:"member" value:"10.1" > label:<name:"ns_instance" value:"10.1.0.40" > label:<name:"servicegroup" value:"perf-prom-prometheus-operator-alertmanager_monitoring_443_perf-prom-prometheus-operator-alertmanager_monitoring_9093_svc" > counter:<value:0 >  was collected before with the same name and label values
* collected metric servicegroup_total_request_bytes label:<name:"member" value:"10.1" > label:<name:"ns_instance" value:"10.1.0.40" > label:<name:"servicegroup" value:"perf-prom-prometheus-operator-alertmanager_monitoring_443_perf-prom-prometheus-operator-alertmanager_monitoring_9093_svc" > counter:<value:0 >  was collected before with the same name and label values
* collected metric servicegroup_total_response_bytes label:<name:"member" value:"10.1" > label:<name:"ns_instance" value:"10.1.0.40" > label:<name:"servicegroup" value:"perf-prom-prometheus-operator-alertmanager_monitoring_443_perf-prom-prometheus-operator-alertmanager_monitoring_9093_svc" > counter:<value:0 >  was collected before with the same name and label values
* collected metric servicegroup_current_client_connections label:<name:"member" value:"10.1" > label:<name:"ns_instance" value:"10.1.0.40" > label:<name:"servicegroup" value:"perf-prom-prometheus-operator-alertmanager_monitoring_443_perf-prom-prometheus-operator-alertmanager_monitoring_9093_svc" > gauge:<value:0 >  was collected before with the same name and label values
* collected metric servicegroup_surge_count label:<name:"member" value:"10.1" > label:<name:"ns_instance" value:"10.1.0.40" > label:<name:"servicegroup" value:"perf-prom-prometheus-operator-alertmanager_monitoring_443_perf-prom-prometheus-operator-alertmanager_monitoring_9093_svc" > gauge:<value:0 >  was collected before with the same name and label values
* collected metric servicegroup_current_server_connections label:<name:"member" value:"10.1" > label:<name:"ns_instance" value:"10.1.0.40" > label:<name:"servicegroup" value:"perf-prom-prometheus-operator-alertmanager_monitoring_443_perf-prom-prometheus-operator-alertmanager_monitoring_9093_svc" > gauge:<value:0 >  was collected before with the same name and label values
* collected metric servicegroup_server_established_connections label:<name:"member" value:"10.1" > label:<name:"ns_instance" value:"10.1.0.40" > label:<name:"servicegroup" value:"perf-prom-prometheus-operator-alertmanager_monitoring_443_perf-prom-prometheus-operator-alertmanager_monitoring_9093_svc" > gauge:<value:0 >  was collected before with the same name and label values
* collected metric servicegroup_current_reuse_pool label:<name:"member" value:"10.1" > label:<name:"ns_instance" value:"10.1.0.40" > label:<name:"servicegroup" value:"perf-prom-prometheus-operator-alertmanager_monitoring_443_perf-prom-prometheus-operator-alertmanager_monitoring_9093_svc" > gauge:<value:0 >  was collected before with the same name and label values
* collected metric servicegroup_max_clients label:<name:"member" value:"10.1" > label:<name:"ns_instance" value:"10.1.0.40" > label:<name:"servicegroup" value:"perf-prom-prometheus-operator-alertmanager_monitoring_443_perf-prom-prometheus-operator-alertmanager_monitoring_9093_svc" > gauge:<value:0 >  was collected before with the same name and label values

My config:

> sh run | grep alertmanager_monitoring_443_perf-prom-prometheus-operator-alertmanager_monitoring_9093_svc
add serviceGroup perf-prom-prometheus-operator-alertmanager_monitoring_443_perf-prom-prometheus-operator-alertmanager_monitoring_9093_svc HTTP -maxClient 0 -maxReq 0 -cip ENABLED NS-Client-IP -usip NO -useproxyport YES -cltTimeout 180 -svrTimeout 360 -CKA NO -TCPB NO -CMP NO
add lb vserver perf-prom-prometheus-operator-alertmanager_monitoring_443_perf-prom-prometheus-operator-alertmanager_monitoring_9093_svc HTTP 0.0.0.0 0 -persistenceType NONE -cltTimeout 180
add cs action perf-prom-prometheus-operator-alertmanager_monitoring_443_perf-prom-prometheus-operator-alertmanager_monitoring_9093_svc -targetLBVserver perf-prom-prometheus-operator-alertmanager_monitoring_443_perf-prom-prometheus-operator-alertmanager_monitoring_9093_svc
add cs policy perf-prom-prometheus-operator-alertmanager_monitoring_443_perf-prom-prometheus-operator-alertmanager_monitoring_9093_svc -rule "HTTP.REQ.HOSTNAME.SERVER.EQ(\"perf-alertmanager.aws.example.dev\") && HTTP.REQ.URL.PATH.SET_TEXT_MODE(IGNORECASE).STARTSWITH(\"/\")" -action perf-prom-prometheus-operator-alertmanager_monitoring_443_perf-prom-prometheus-operator-alertmanager_monitoring_9093_svc
bind lb vserver perf-prom-prometheus-operator-alertmanager_monitoring_443_perf-prom-prometheus-operator-alertmanager_monitoring_9093_svc perf-prom-prometheus-operator-alertmanager_monitoring_443_perf-prom-prometheus-operator-alertmanager_monitoring_9093_svc
bind cs vserver perf-172.19.0.88_443_ssl -policyName perf-prom-prometheus-operator-alertmanager_monitoring_443_perf-prom-prometheus-operator-alertmanager_monitoring_9093_svc -priority 200000004
bind serviceGroup perf-prom-prometheus-operator-alertmanager_monitoring_443_perf-prom-prometheus-operator-alertmanager_monitoring_9093_svc 10.1.1.27 30903
bind serviceGroup perf-prom-prometheus-operator-alertmanager_monitoring_443_perf-prom-prometheus-operator-alertmanager_monitoring_9093_svc 10.1.1.148 30903
bind serviceGroup perf-prom-prometheus-operator-alertmanager_monitoring_443_perf-prom-prometheus-operator-alertmanager_monitoring_9093_svc 10.1.1.152 30903

[rokett]

This is fixed in the v4.3.1 by adding the member port as a label.