blackbox exporter failing to recognised the service is not responding

[sudhirpandey]

I was quite amazed to see that the service is termed still up and ok while in the stdout logs of the blackbox exporter is showing errors like this

time="2016-11-21T10:46:40Z" level=warning msg="Error for HTTP request to http://svc.proj.svc:19080: Get http://svc.proj.svc:19080: dial tcp4 172.30.162.209:19080: getsockopt: no route to host" source="http.go:159"

I have intentionally , scaled a service to zero pods in openshift origin. and i was expecting a instance down alert from prometheus.. How ever form target view the blackbox exporter never seems to recognize that service is down and is not responding

while i try to directly query the backend service using the blackbox exportetservice ip

curl http://172.30.235.19/probe?target=svc.proj.svc:19080
probe_ip_protocol 4
probe_http_status_code 0
probe_http_content_length 0
probe_http_redirects 0
probe_http_ssl 0
probe_duration_seconds 3.031048
probe_success 0

i am using module http_2xx so since the status code is 0 , should not it have raised (lavel) service as down in targets view in prometheus Web UI.

Also additionally this query works

curl http://172.30.235.19/probe?target=svc.project.svc:19080&module=http_2xx

probe_http_status_code 0
probe_http_content_length 0
probe_http_redirects 0
probe_http_ssl 0
probe_duration_seconds 3.040584
probe_success 0

while

 curl http://172.30.235.19/probe?module=http_2xx&target=svc.project.svc:19080

produces
Target parameter is missing

and bails out .

Am i missing something on the alert part. if a service dont have any endpoints i am hoping that probing to it fails or i am missing something?

I am using blackbox exporter container version0.3,0
as it start it shows

"Starting blackbox_exporter (version=0.3.0, branch=master, revision=43cfa9854e595993b64925c8a5b5f39086b03dae)"

[brian-brazil]

Those are showing probe_success as 0 as is expected when the target is down.

Target parameter is missing

That's due to bash interpreting the & as the end of the command. Quote the URL.

[sudhirpandey]

But the target is down and is not replying(reachable) as we can see from pod logs
and probe_http_status_code 0 is as well not 200 ,
so probe sucess is 0

Then should not i get a error as its down in prometheus web UI. The State is still UP while the service is down and not replying.

[brian-brazil]

I think you should read https://www.robustperception.io/whats-up-doc/, in particular the last paragraph.

[sudhirpandey]

Thanks for more info, but for particular service that is down ..right now .. i am still getting
up=1
Just a bit confused that it should at least have been up=0 and if it was not sufficient , we could have employ the other sources to make sure of it.

[brian-brazil]

That's correct. The blackbox exporter is being successfully scraped. You need to look at probe_success to see if the probe worked.

[sudhirpandey]

Thanks 👍 .
I used the scrapped metric and it work as expected now. Was a bit confusing though, thanks for the help :).

[vishalsparmar]

That's correct. The blackbox exporter is being successfully scraped. You need to look at probe_success to see if the probe worked.

@brian-brazil We have prometheus installed and also blackboxexporter via helm. Post defining servicemonitor for Blackboxexporter i am able to see my blackboxexporter service pods in Prometheus Target but its not probing the targets added. screen shot for prometheus discovery
can you advise whats missed here please. Thanks