TLS & User/Pass auth accessing druid coordinator endpoint

[scotforescout]

I didn't see an option to add in a username & password & specify a cacert when accessing a druid coordinator endpoint. Any chance to add those?

[iamabhishek-dubey]

Nice feature request, I guess we can work on this

[scotforescout]

Tried it out today, I'm only getting druid health check status being returned, not seeing the other ones now.

curl http://x.x.x.x:8080/metrics | grep druid
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
HELP druid_health_status Health of Druid, 1 is healthy 0 is not
TYPE druid_health_status counter
druid_health_status{druid="health"} 1
100 6338 0 6338 0 0 44321 0 --:--:-- --:--:-- --:--:-- 44321

[iamabhishek-dubey]

I have just tested and it seems to be working fine

# TYPE druid_supervisors gauge
druid_supervisors{healthy="false",state="UNHEALTHY_TASKS",supervisor_name="Test"} 1
druid_supervisors{healthy="false",state="UNHEALTHY_TASKS",supervisor_name="Test2"} 1
# HELP druid_tasks_duration Druid tasks duration and state
# TYPE druid_tasks_duration gauge

Can you confirm that you have some running tasks in druid?

Also, have you configured the emitter in your druid
https://github.com/opstree/druid-exporter#druid-configuration-changes

If you have done both things and still you don't have metrics, please share the logs.

[scotforescout]

We wrote our update on the ultils/http.go part, by modifying the requests to allow the basic auth (and ignoring TLS for the time being) We are able to get metrics that way:

package utils

import (
"druid-exporter/logger"
"github.com/go-kit/kit/log/level"
"io/ioutil"
"net/http"
"crypto/tls"
"os"
)

var authUser string
var authPass string

func init() {

authUser = os.Getenv("AUTH_USER")
authPass = os.Getenv("AUTH_PASS")

}

// GetHealth returns that druid is healthy or not
func GetHealth(url string) float64 {
druidLogger := logger.GetLoggerInterface()
tr := &http.Transport{
TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
}
client := &http.Client{ Transport: tr}
req, err := http.NewRequest("GET", url, nil)
if err != nil {
level.Error(druidLogger).Log("msg", "Cannot create GET request for druid healthcheck", "err", err)
}
req.SetBasicAuth(authUser, authPass)
resp, err := client.Do(req)
if err != nil {
level.Error(druidLogger).Log("msg", "Error while making GET request for druid healthcheck", "err", err)
}
level.Info(druidLogger).Log("msg", "GET request is successful on druid healthcheck", "url", url)
defer resp.Body.Close()
if resp.StatusCode == 200 {
return 1
}

return 0
}

// GetResponse will return API response for druid
func GetResponse(url string, queryType string) ([]byte, error) {
druidLogger := logger.GetLoggerInterface()
tr := &http.Transport{
TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
}
client := &http.Client{ Transport: tr}
req, err := http.NewRequest("GET", url, nil)
if err != nil {
level.Error(druidLogger).Log("msg", "Cannot create http request", "err", err)
}
req.SetBasicAuth(authUser, authPass)

resp, err := client.Do(req)
if err != nil {
level.Error(druidLogger).Log("msg", "Error while making http request", "err", err)
}

defer resp.Body.Close()
level.Info(druidLogger).Log("msg", "GET request is successful for druid api", "url", url)

return ioutil.ReadAll(resp.Body)
}

While running the newest update from here... we show that the druid exporter running in AWS ECS and we have debug logs turned on with this output

2020-06-08 23:29:14{"level":"info","msg":"GET request is successful for druid api","ts":"2020-06-09T06:29:14.317Z","url":"ELB_OMITTED:9088/druid/coordinator/v1/datasources?simple"}
2020-06-08 23:29:14{"level":"info","msg":"Successfully retrieved the data for druid segment","ts":"2020-06-09T06:29:14.317Z"}
2020-06-08 23:29:14{"level":"info","msg":"GET request is successful on druid healthcheck","ts":"2020-06-09T06:29:14.246Z","url":"ELB_OMITTED:9088/status/health"}
2020-06-08 23:29:14{"level":"info","msg":"Successfully retrieved the data for druid healthcheck","ts":"2020-06-09T06:29:14.229Z"}
2020-06-08 23:28:59{"level":"info","msg":"GET request is successful for druid api","ts":"2020-06-09T06:28:59.511Z","url":"ELB_OMITTED:9088/druid/indexer/v1/supervisor?full"}
2020-06-08 23:28:59{"level":"info","msg":"Successfully retrieved the data for druid's supervisors tasks","ts":"2020-06-09T06:28:59.511Z"}
2020-06-08 23:28:59{"level":"info","msg":"Successfully retrieved the data for druid's tasks","ts":"2020-06-09T06:28:59.442Z"}

I'll dig into this more tomorrow.

Please do know that we truly do appreciate the work and how fast you were able to turn around and get this feature request in!

[scotforescout]

Dug into it more, this is not an entire answer but you should be able to see the issue if you add in debugging more on the response data.

https://github.com/opstree/druid-exporter/blob/master/collector/druid.go#L35
https://github.com/opstree/druid-exporter/blob/master/collector/druid.go#L50
https://github.com/opstree/druid-exporter/blob/master/collector/druid.go#L65

While the requests are being authenticated, the entire response isn't being reported back with all the stats, it's just reporting back an array of values and when json.Unmarshal(responseData, &metric) it nullifies the data making it through.

[scotforescout]

Error was on our side, many different things are embedded and after breaking things down we found it was an issue on the password being stored.