Kubernetes ingress discovery sets invalid scheme

[nailgun]

Bug Report

What did you do?
I have a single ingress object with multiple backends. One is https, second is http.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    prometheus.io/path: healthz
  name: web-v1
  namespace: myns
spec:
  rules:
  - host: api.smp.io
    http:
      paths:
      - backend:
          serviceName: web-v1
          servicePort: 80
        path: /apps/v1/
  - host: smp.ingress.svc.cluster.local
    http:
      paths:
      - backend:
          serviceName: web-v1
          servicePort: 80
        path: /apps/v1/
  tls:
  - hosts:
    - api.smp.io
    secretName: api.smp.io-tls

What did you expect to see?
api.smp.io - httpS
smp.ingress.svc.cluster.local - http

Ingress controller (in my case - nginx) works this way.

What did you see instead? Under which circumstances?
Prometheus sets __meta_kubernetes_ingress_scheme to https for both backends.

Environment

• System information: GKE cluster
• Prometheus version: v2.3.1
• Prometheus configuration file:

    # Scrape config for probing ingresses via the Blackbox Exporter
    - job_name: ingress
      kubernetes_sd_configs:
      - role: ingress

      metrics_path: /probe
      params:
        module: [http_2xx]

      scrape_timeout: 30s

      relabel_configs:
      - source_labels: [__meta_kubernetes_ingress_annotation_prometheus_io_probe]
        action: drop
        regex: false
      - source_labels: [__meta_kubernetes_ingress_path]
        action: drop
        regex: /.well-known/acme-challenge
      - source_labels: [__meta_kubernetes_ingress_scheme, __address__, __meta_kubernetes_ingress_path, __meta_kubernetes_ingress_annotation_prometheus_io_path]
        regex: (.+);(.+);(.+);(.*)
        replacement: ${1}://${2}${3}${4}
        target_label: __param_target
      - source_labels: [__meta_kubernetes_ingress_annotation_prometheus_io_blackbox_module]
        action: replace
        target_label: __param_module
      - target_label: __address__
        replacement: blackbox-exporter
      - source_labels: [__param_target]
        target_label: instance
      - action: labelmap
        regex: __meta_kubernetes_ingress_label_(.+)
      - source_labels: [__meta_kubernetes_namespace]
        action: replace
        target_label: kubernetes_namespace
      - source_labels: [__meta_kubernetes_ingress_name]
        action: replace
        target_label: kubernetes_ingress_name

Resolution

I found issue here

prometheus/discovery/kubernetes/ingress.go

Lines 179 to 194 in 6eab4bb

	schema := "http"
	if ingress.Spec.TLS != nil {
	schema = "https"
	}
	for _, rule := range ingress.Spec.Rules {
	paths := pathsFromIngressRule(&rule.IngressRuleValue)
	for _, path := range paths {
	tg.Targets = append(tg.Targets, model.LabelSet{
	model.AddressLabel: lv(rule.Host),
	ingressSchemeLabel: lv(schema),
	ingressHostLabel: lv(rule.Host),
	ingressPathLabel: lv(path),
	})
	}
	}

Actually it should check if rule.Host is in one of ingress.Spec.TLS configs.

I'm going to send a PR if project maintainer will confirm that this should be fixed. For now I just override __meta_kubernetes_ingress_scheme in config:

      - source_labels: [__address__]  # fix the bug
        regex: smp.ingress.svc.cluster.local
        replacement: http
        target_label: __meta_kubernetes_ingress_scheme

[krasi-georgiev]

@nailgun thanks for the report, would be appreciated if you have the time to open a PR with a a unit test?

[nailgun]

Yes, maybe on weekend

[krasi-georgiev]

Appreciated 👍

[nailgun]

Here it is. @krasi-georgiev is it better to fix it in same PR/branch or create another one?

[nailgun]

I thought it it better to do it in the same PR. PR has been updated.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.