elasticsearch_cluster_health_status metric is missing

[Xat59]

Hi !

I am using your exporter with these arguments :

--es.uri localhost:9200 \
--web.listen-address :9108 \
--web.telemetry-path /metrics \
--es.all \
--es.indices \
--es.shards  \
--es.snapshots  \
--es.indices_settings  \
--es.timeout 5s 

When querying the metrics page via :9108/metrics I have some missing metrics, such as elasticsearch_cluster_health_status.

Here is the full output :

# HELP elasticsearch_cluster_health_json_parse_failures Number of errors while parsing JSON.
# TYPE elasticsearch_cluster_health_json_parse_failures counter
elasticsearch_cluster_health_json_parse_failures 0
# HELP elasticsearch_cluster_health_total_scrapes Current total ElasticSearch cluster health scrapes.
# TYPE elasticsearch_cluster_health_total_scrapes counter
elasticsearch_cluster_health_total_scrapes 81
# HELP elasticsearch_cluster_health_up Was the last scrape of the ElasticSearch cluster health endpoint successful.
# TYPE elasticsearch_cluster_health_up gauge                       
elasticsearch_cluster_health_up 0         
# HELP elasticsearch_clusterinfo_last_retrieval_failure_ts Timestamp of the last failed cluster info retrieval
# TYPE elasticsearch_clusterinfo_last_retrieval_failure_ts gauge                        
elasticsearch_clusterinfo_last_retrieval_failure_ts{url="localhost:9200"} 1.619095367e+09
# HELP elasticsearch_clusterinfo_up Up metric for the cluster info collector
# TYPE elasticsearch_clusterinfo_up gauge                        
elasticsearch_clusterinfo_up{url="localhost:9200"} 0
# HELP elasticsearch_exporter_build_info A metric with a constant '1' value labeled by version, revision, branch, and goversion from which elasticsearch_exporter was built.
# TYPE elasticsearch_exporter_build_info gauge
elasticsearch_exporter_build_info{branch="master",goversion="go1.12.3",revision="fe20e499ffdd6053e6153bac15eae494e08931df",version="1.1.0"} 1
# HELP elasticsearch_index_stats_json_parse_failures Number of errors while parsing JSON.
# TYPE elasticsearch_index_stats_json_parse_failures counter
elasticsearch_index_stats_json_parse_failures 0
# HELP elasticsearch_index_stats_total_scrapes Current total ElasticSearch index scrapes.
# TYPE elasticsearch_index_stats_total_scrapes counter
elasticsearch_index_stats_total_scrapes 76
# HELP elasticsearch_index_stats_up Was the last scrape of the ElasticSearch index endpoint successful.
# TYPE elasticsearch_index_stats_up gauge
elasticsearch_index_stats_up 0
# HELP elasticsearch_indices_settings_stats_json_parse_failures Number of errors while parsing JSON.
# TYPE elasticsearch_indices_settings_stats_json_parse_failures counter
elasticsearch_indices_settings_stats_json_parse_failures 0
# HELP elasticsearch_indices_settings_stats_read_only_indices Current number of read only indices within cluster
# TYPE elasticsearch_indices_settings_stats_read_only_indices gauge
elasticsearch_indices_settings_stats_read_only_indices 0
# HELP elasticsearch_indices_settings_stats_total_scrapes Current total ElasticSearch Indices Settings scrapes.
# TYPE elasticsearch_indices_settings_stats_total_scrapes counter
elasticsearch_indices_settings_stats_total_scrapes 76
# HELP elasticsearch_indices_settings_stats_up Was the last scrape of the ElasticSearch Indices Settings endpoint successful.
# TYPE elasticsearch_indices_settings_stats_up gauge
elasticsearch_indices_settings_stats_up 0
# HELP elasticsearch_node_stats_json_parse_failures Number of errors while parsing JSON.
# TYPE elasticsearch_node_stats_json_parse_failures counter
elasticsearch_node_stats_json_parse_failures 0
# HELP elasticsearch_node_stats_total_scrapes Current total ElasticSearch node scrapes.
# TYPE elasticsearch_node_stats_total_scrapes counter
elasticsearch_node_stats_total_scrapes 76
# HELP elasticsearch_node_stats_up Was the last scrape of the ElasticSearch nodes endpoint successful.
# TYPE elasticsearch_node_stats_up gauge
elasticsearch_node_stats_up 0
# HELP elasticsearch_snapshot_stats_json_parse_failures Number of errors while parsing JSON.
# TYPE elasticsearch_snapshot_stats_json_parse_failures counter
elasticsearch_snapshot_stats_json_parse_failures 0
# HELP elasticsearch_snapshot_stats_total_scrapes Current total ElasticSearch snapshots scrapes.
# TYPE elasticsearch_snapshot_stats_total_scrapes counter
elasticsearch_snapshot_stats_total_scrapes 76
# HELP elasticsearch_snapshot_stats_up Was the last scrape of the ElasticSearch snapshots endpoint successful.
# TYPE elasticsearch_snapshot_stats_up gauge
elasticsearch_snapshot_stats_up 0
# HELP go_gc_duration_seconds A summary of the GC invocation durations.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 2.86e-05
go_gc_duration_seconds{quantile="0.25"} 4.1e-05
go_gc_duration_seconds{quantile="0.5"} 0.000228098
go_gc_duration_seconds{quantile="0.75"} 0.000542796
go_gc_duration_seconds{quantile="1"} 0.00141429
go_gc_duration_seconds_sum 0.005662959
go_gc_duration_seconds_count 18
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 15
# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 3.04848e+06
# HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
# TYPE go_memstats_alloc_bytes_total counter
go_memstats_alloc_bytes_total 5.5182024e+07
# HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.
# TYPE go_memstats_buck_hash_sys_bytes gauge
go_memstats_buck_hash_sys_bytes 1.446097e+06
# HELP go_memstats_frees_total Total number of frees.
# TYPE go_memstats_frees_total counter
go_memstats_frees_total 132885
# HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata.
# TYPE go_memstats_gc_sys_bytes gauge
go_memstats_gc_sys_bytes 2.377728e+06
# HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use.
# TYPE go_memstats_heap_alloc_bytes gauge
go_memstats_heap_alloc_bytes 3.04848e+06
# HELP go_memstats_heap_idle_bytes Number of heap bytes waiting to be used.
# TYPE go_memstats_heap_idle_bytes gauge
go_memstats_heap_idle_bytes 6.2562304e+07
# HELP go_memstats_heap_inuse_bytes Number of heap bytes that are in use.
# TYPE go_memstats_heap_inuse_bytes gauge
go_memstats_heap_inuse_bytes 3.858432e+06
# HELP go_memstats_heap_objects Number of allocated objects.
# TYPE go_memstats_heap_objects gauge
go_memstats_heap_objects 9852
# HELP go_memstats_heap_released_bytes_total Total number of heap bytes released to OS.
# TYPE go_memstats_heap_released_bytes_total counter
go_memstats_heap_released_bytes_total 0
# HELP go_memstats_heap_sys_bytes Number of heap bytes obtained from system.
# TYPE go_memstats_heap_sys_bytes gauge
go_memstats_heap_sys_bytes 6.6420736e+07
# HELP go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection.
# TYPE go_memstats_last_gc_time_seconds gauge
go_memstats_last_gc_time_seconds 1.6190955537775931e+09
# HELP go_memstats_lookups_total Total number of pointer lookups.
# TYPE go_memstats_lookups_total counter
go_memstats_lookups_total 0
# HELP go_memstats_mallocs_total Total number of mallocs.
# TYPE go_memstats_mallocs_total counter
go_memstats_mallocs_total 142737
# HELP go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures.
# TYPE go_memstats_mcache_inuse_bytes gauge
go_memstats_mcache_inuse_bytes 3472
# HELP go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system.
# TYPE go_memstats_mcache_sys_bytes gauge
go_memstats_mcache_sys_bytes 16384
# HELP go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures.
# TYPE go_memstats_mspan_inuse_bytes gauge
go_memstats_mspan_inuse_bytes 30960
# HELP go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system.
# TYPE go_memstats_mspan_sys_bytes gauge
go_memstats_mspan_sys_bytes 49152
# HELP go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place.
# TYPE go_memstats_next_gc_bytes gauge
go_memstats_next_gc_bytes 4.194304e+06
# HELP go_memstats_other_sys_bytes Number of bytes used for other system allocations.
# TYPE go_memstats_other_sys_bytes gauge
go_memstats_other_sys_bytes 763943
# HELP go_memstats_stack_inuse_bytes Number of bytes in use by the stack allocator.
# TYPE go_memstats_stack_inuse_bytes gauge
go_memstats_stack_inuse_bytes 688128
# HELP go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator.
# TYPE go_memstats_stack_sys_bytes gauge
go_memstats_stack_sys_bytes 688128
# HELP go_memstats_sys_bytes Number of bytes obtained by system. Sum of all system allocations.
# TYPE go_memstats_sys_bytes gauge
go_memstats_sys_bytes 7.1762168e+07
# HELP http_request_duration_microseconds The HTTP request latencies in microseconds.
# TYPE http_request_duration_microseconds summary
http_request_duration_microseconds{handler="prometheus",quantile="0.5"} 2147.683
http_request_duration_microseconds{handler="prometheus",quantile="0.9"} 3518.972
http_request_duration_microseconds{handler="prometheus",quantile="0.99"} 3619.371
http_request_duration_microseconds_sum{handler="prometheus"} 178060.782
http_request_duration_microseconds_count{handler="prometheus"} 75
# HELP http_request_size_bytes The HTTP request sizes in bytes.
# TYPE http_request_size_bytes summary
http_request_size_bytes{handler="prometheus",quantile="0.5"} 214
http_request_size_bytes{handler="prometheus",quantile="0.9"} 214
http_request_size_bytes{handler="prometheus",quantile="0.99"} 214
http_request_size_bytes_sum{handler="prometheus"} 12752
http_request_size_bytes_count{handler="prometheus"} 75
# HELP http_requests_total Total number of HTTP requests made.
# TYPE http_requests_total counter
http_requests_total{code="200",handler="prometheus",method="get"} 75
# HELP http_response_size_bytes The HTTP response sizes in bytes.
# TYPE http_response_size_bytes summary
http_response_size_bytes{handler="prometheus",quantile="0.5"} 2063
http_response_size_bytes{handler="prometheus",quantile="0.9"} 10692
http_response_size_bytes{handler="prometheus",quantile="0.99"} 10700
http_response_size_bytes_sum{handler="prometheus"} 343527
http_response_size_bytes_count{handler="prometheus"} 75
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 0.26
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 1024
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 8
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 1.2976128e+07
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.61909476672e+09
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 1.15130368e+08

As you can see, I do not have elasticsearch_cluster_health_status, and some other metrics I was expecting.

Thanks a lot.

[bartjolenovation]

I don't see that you use the es.cluster_settings true option (it defaults to false), did you set that?

[Xat59]

I just appended es.cluster_settings as a new parameter for the exporter, then restarted the exporter.
But I have not elasticsearch_cluster_health_status metric yet.

By the way, I assume that es.cluster_settings is the same than ``es.cluster_settings true`. Am I right ?

[bartjolenovation]

I don't believe so, I think you do need to add the true part. Ow and I think I got the syntax wrong, I think it needs to be --es.cluster_settings=true

[Xat59]

Ok so in complement of adding --es.cluster_settings, I notice that we must specify the scheme in --es.uri even if the uri uses http scheme :

Apr 22 13:04:45 XCLELKP1 elasticsearch_exporter[32882]: level=error ts=2021-04-22T13:04:45.133094266Z caller=clusterinfo.go:253 msg="failed to get cluster info" err="Get localhost:9200: unsupported protocol scheme \"localhost\""
Apr 22 13:04:45 XCLELKP1 elasticsearch_exporter[32882]: level=error ts=2021-04-22T13:04:45.133142466Z caller=clusterinfo.go:174 msg="failed to retrieve cluster info from ES" err="Get localhost:9200: unsupported protocol scheme \"localhost\""

So I replaced --es.uri localhost:9200 with --es.uri http://localhost:9200.

Thanks a lot @bartjolenovation , my problem is solved.

[han916701843]

I don't believe so, I think you do need to add the true part. Ow and I think I got the syntax wrong, I think it needs to be --es.cluster_settings=true

I add --es.cluster_settings=true parameter,then get error "unexpected true", as below