Kubernetes SD; Support external name services

[Chronojam]

This is a proposal to add support for external name services to the Kubernetes SD

Given the following kubernetes service configuration:

kind: Service
apiVersion: v1
metadata:
  name: my-service
  annotations:
    "prometheus.io/scrape": "true"
spec:
  type: ExternalName
  externalName: my.service.example.com

and an endpoint discoverer:

kubernetes_sd_configs:
  - role: endpoints

I think the Kubernetes SD should create 1 scrape per DNS type
I dont think there is any need to support other types than A/AAAA and SRV (similar to the dns discoverer)?

my.service.example.com:AAAA => 10.0.0.1
my.service.example.com:SRV1 => 10.0.0.1:8080
my.service.example.com:SRV2 => 10.0.0.2:8080

for each endpoint discovered, we should attach a __meta_kubernetes_service_dnstype label with
type ("AAAA"/"SRV")

[brian-brazil]

How are you envisioning scraping these?

This sounds more like a load balancer rather than the individual targets we'd want to scrape.

[Chronojam]

An example might be an external etcd cluster (which ive just grabbed from the clustering guide);

$ dig +noall +answer SRV _etcd-server._tcp.example.com
_etcd-server._tcp.example.com. 300 IN  SRV  0 0 2380 infra0.example.com.
_etcd-server._tcp.example.com. 300 IN  SRV  0 0 2380 infra1.example.com.
_etcd-server._tcp.example.com. 300 IN  SRV  0 0 2380 infra2.example.com.

I might setup my service like this

kind: Service
apiVersion: v1
metadata:
  name: my-external-etcd
  annotations:
    "prometheus.io/scrape": "true"
spec:
  type: ExternalName
  externalName: _etcd-server-ssl._tcp.example.com

Then the created job would monitor each record there (like the DNS discoverer does)

Job: my-external-etcd
endpoint: infra0.example.com:2380/metrics
endpoint: infra1.example.com:2380/metrics

[crandles]

This seems better suited for the service discoverer (role: service), since there are no endpoints to discover for this.

The dns_sd_config allows for specifying the type of DNS record to look for, would it make sense to add that option here, as well as a refresh_interval? It does not seem like any options are role-specific, so this seems sub-optimal.

Would it be viable to simply try resolving a SRV record, and falling back to the externalName entry (instead of trying to resolve its IP)?

[discordianfish]

Just tried this and hoped it would just work but it doesn't. Seems sensible to me to support this. It's IMO the right way to scrape anything outside of a kubernetes cluster.

[simonpasquier]

This should be fixed since v2.7.0 by de6f3b6. @discordianfish which version did you try?

[brancz]

@simonpasquier I don't think so. I've thought about this a bit more and I don't think an external service type service is generalizable. We still want to perform whitebox monitoring, and we can't generalize that for the same reason we have multiple service discovery mechanisms. Those external systems simply are not part of the same service discovery infrastructure.

[brian-brazil]

Those are my thoughts too. Anything outside of k8 will be monitored the ways you'd monitor anything else outside of k8.