Kubernetes ingress discovery does not set address label for ingresses without host

[Mattias-]

What did you do?
Configured discovery of kubernetes ingresses.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: alertmanager-ingress
  namespace: monitoring-prometheus
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/rewrite-target: "/"
    prometheus.io/probe: "true"
spec:
  rules:
    - http:
        paths:
          - path: /alertmanager
            backend:
              serviceName: alertmanager
              servicePort: http
    - http:
        paths:
          - path: /alertmanager
            backend:
              serviceName: alertmanager
              servicePort: http
      host: alertmanager.local

What did you expect to see?
All annotated ingresses to show up as targets that can be successfully scraped.

What did you see instead? Under which circumstances?
No target label is set for ingresses that hasn't specified a host and the scraping fail.

Environment

• System information:

Linux 4.9.64 x86_64

• Prometheus version:

2.2.1 docker image

• Prometheus configuration file:

    - job_name: 'kubernetes-ingresses'

      metrics_path: /probe
      params:
        module: [http_2xx]

      kubernetes_sd_configs:
      - role: ingress

      relabel_configs:
      - source_labels: [__meta_kubernetes_ingress_annotation_prometheus_io_probe]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_ingress_scheme,__address__,__meta_kubernetes_ingress_path]
        regex: (.+);(.+);(.+)
        replacement: ${1}://${2}${3}
        target_label: __param_target
      - source_labels: [__meta_kubernetes_ingress_scheme,__meta_kubernetes_ingress_path]
        regex: (.+);(.+)
        replacement: ${1}://xx${2}
        target_label: mylabel
      - target_label: __address__
        replacement: blackbox-exporter:9115
      - source_labels: [__param_target]
        target_label: instance
      - action: labelmap
        regex: __meta_kubernetes_ingress_label_(.+)
      - source_labels: [__meta_kubernetes_namespace]
        target_label: kubernetes_namespace
      - source_labels: [__meta_kubernetes_ingress_name]
        target_label: kubernetes_name

[brian-brazil]

What address label did you expect to be set here?

[Mattias-]

The IP that the ingress controller has allocated to this ingress object to fulfill it. In the case below 192.168.122.106.

I can see general problems with ingress discovery as there's no guarantee that either the hostname used in the Host header resolves to something useful or that the allocated IP address is reachable from inside the k8s cluster.

$ kubectl --namespace=monitoring-prometheus describe ingress alertmanager-ingress 
Name:             alertmanager-ingress
Namespace:        monitoring-prometheus
Address:          192.168.122.106
Default backend:  default-http-backend:80 (172.17.0.5:8080)
Rules:
  Host                Path  Backends
  ----                ----  --------
  *                   
                      /alertmanager   alertmanager:http (<none>)
  alertmanager.local  
                         alertmanager:http (<none>)
Annotations:
Events:
  Type    Reason  Age              From                      Message
  ----    ------  ----             ----                      -------
  Normal  UPDATE  3m (x9 over 2d)  nginx-ingress-controller  Ingress monitoring-prometheus/alertmanager-ingress

$ kubectl --namespace=monitoring-prometheus get ingress alertmanager-ingress  --output yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"kubernetes.io/ingress.class":"nginx","nginx.ingress.kubernetes.io/rewrite-target":"/","prometheus.io/probe":"true"},"name":"alertmanager-ingress","namespace":"monitoring-prometheus"},"spec":{"rules":[{"http":{"paths":[{"backend":{"serviceName":"alertmanager","servicePort":"http"},"path":"/alertmanager"}]}},{"host":"alertmanager.local","http":{"paths":[{"backend":{"serviceName":"alertmanager","servicePort":"http"}}]}}]}}
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/rewrite-target: /
    prometheus.io/probe: "true"
  creationTimestamp: 2018-03-22T21:47:28Z
  generation: 7
  name: alertmanager-ingress
  namespace: monitoring-prometheus
  resourceVersion: "76373"
  selfLink: /apis/extensions/v1beta1/namespaces/monitoring-prometheus/ingresses/alertmanager-ingress
  uid: 9e03f903-2e1a-11e8-930e-f8dac8f5e6eb
spec:
  rules:
  - http:
      paths:
      - backend:
          serviceName: alertmanager
          servicePort: http
        path: /alertmanager
  - host: alertmanager.local
    http:
      paths:
      - backend:
          serviceName: alertmanager
          servicePort: http
status:
  loadBalancer:
    ingress:
    - ip: 192.168.122.106

[yunlzheng]

same issue