Kubernetes service auto-discovery expect services available at *.svc

[bkupidura]

Kubernetes service auto-discovery expect to have service available at SVC_NAME.SVC_NAMESPACE.svc this is not always true.

$ host nova-api.namespace.svc
Host nova-api.namespace.svc not found: 3(NXDOMAIN)
$ host nova-api.namespace.svc.cluster.local
nova-api.namespace.svc.cluster.local has address 10.233.56.241

User should have possibility to somehow provide domain suffix.

I tried to use relabel_configs, but without success:
- source_labels: [__param_target]
target_label: instance
regex: (.+):(\d+)
replacement: $1.cluster.local:$2
action: replace

Please check: https://github.com/prometheus/prometheus/blob/master/retrieval/discovery/kubernetes/service.go#L115

[fabxc]

@jimmidyson ideas how to handle that? Should we just let users specify it or are you aware of any API endpoint we could dynamically get that info from?

[jimmidyson]

There is no way I know of to get the domain, but the cluster domain should be added as a search suffix to resolv.conf (e.g. search cluster.local). That's why the short name is recommended for cluster portability. Resolving the short name does rely on the resolver supporting search suffixes, afaik that is only a problem with stuff using the older musl libc (e.g. Alpine <= 3.3).

I guess you could add an optional domain to Prometheus config.

[fabxc]

Mh, I'd like to avoid adding more options and what you say sounds totally correct.
@bkupidura is that a viable option for you and if not, why?

[bkupidura]

Yeah, fixing this in resolv.conf sounds reasonable. We can close this issue, Thanks!

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.