[Exporter] How to use the blackbox_exporter with prometheus-operator?

[galexrt]

The blackbox_exporter requires targets and params to be set for the exporter to report data.
See the example configuration for prometheus: https://github.com/prometheus/blackbox_exporter#prometheus-configuration

Is there a way to achieve a config like the above with prometheus-operator and how?
If it is achievable, it would be nice to add it to the user-guides, as I'm definitely not the only one interested in running the blackbox_exporter.

[fabxc]

We ultimately want to have blackbox probing fully-integrated. Right now you'd have to achieve this by using a custom Prometheus configuration with your operator. You can do so by omitting the serviceMonitorSelector.

For a deeper integration, we have to understand the different use cases better. Can you elaborate a bit on which things you want to blackbox probe?

[galexrt]

Thanks for the hints at using a custom configuration for this!

I was thinking about a) cluster internal and b) external blackbox probing (tcp, icmp, http, https).

[ohaiwalt]

Also interested in this, and we're currently using blackbox exporter to monitor availability for external services to give us a bit of a clearer picture of what a user is seeing until the rest of our monitoring shapes up.

Current workaround is having a second Prometheus instance with a custom configuration that sends to our alertmanager.

[MattMencel]

Me too...

I'm trying to figure out how to monitor things external to the k8s cluster.

I.E. WMI exporter and Node Exporter for nodes in our regular VM environment, and snmp and blackbox exporter for ping/snmp monitoring of other devices on our network.

[brancz]

@MattMencel anything not running as part of the Kubernetes cluster or in it is a use case for a custom Prometheus configuration. You can specify your own configuration by skipping the serviceMonitorSelector. I've been meaning to write some documentation on custom configurations, I'm hoping to do this somewhat soon.

[MattMencel]

Yeah that's where I got stuck. I'm still pretty new to prometheus and k8s.

For now I've got an external prometheus VM where I'm doing some of this monitoring. Eventually I'm wanting to move it all into the k8s cluster.

Thanks!

[gianrubio]

I'm starting to monitor external services with blackbox and operator so I got onto this issue.

My proposal for this issue and for others that will not be managed by operator is to accept a job append configuration. The appendJobSelector will be attached to the end of the job configuration, generated by prometheus. Below are the prometheus spec and the job configmap.

Prometheus

  apiVersion: monitoring.coreos.com/v1alpha1
  kind: Prometheus
  metadata:
    labels:
      prometheus: k8s
    name: k8s
    namespace: monitoring
  spec:
    alerting:
      alertmanagers:
      - name: alertmanager-main
        namespace: monitoring
        port: web
    appendJobSelector:                               
      matchLabels:                              
        name: prometheus-custom-jobs          
        prometheus: k8s                                    
    replicas: 1
  ...

Configmap

kind: ConfigMap
apiVersion: v1
metadata:
  name: prometheus-custom-jobs
  labels:
    name: prometheus-custom-jobs
    prometheus: k8s
data:
    - job_name: 'blackbox'
    metrics_path: /probe
    params:
      module: [http_2xx]  # Look for a HTTP 200 response.
    static_configs:
      - targets:
        - http://prometheus.io    # Target to probe with http.
        - https://prometheus.io   # Target to probe with https.
        - http://example.com:8080 # Target to probe with http on port 8080.
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 127.0.0.1:9115  # Blackbox exporter

[grrywlsn]

@gianrubio +1 for the appendJob idea; that's exactly what I'm looking for.

I like the simplicity of ServiceMonitors and would like to use them in most cases, but there are a few jobs in my existing (non-Operator) Prometheus I'd like to carry across, such as monitoring the external etcd used by Kube masters, which can't be expressed as a ServiceMonitor.

[brancz]

@grrywlsn actually in that case what you want to do is create an Endpoints object yourself. As this is a common theme for etcd, we've prepared an example for it: https://github.com/coreos/prometheus-operator/blob/master/contrib/kube-prometheus/manifests/etcd/etcd-bootkube-gce.yaml

This issue is about actually blackbox probing your applications. For example performing real HTTP requests on your application to see whether it is responding the way you expect.

[brancz]

@gianrubio the point of the Prometheus Operator is to provide sensible abstractions for Prometheus. Your suggestion starts a bit too low level. We want to build abstractions that are higher up, so people don't need to know the Prometheus/blackbox exporter configuration paradigms.

[grrywlsn]

@brancz awesome, thanks for that example! it wasn't all that obvious from the docs that I could see.

[brancz]

@grrywlsn docs contributions are always welcome! 🙂 If anything is not as clear as it should be we're happy about PRs!

[grrywlsn]

Good to know. I'm working on moving our Prometheus deployments to Prometheus Operator deployments at the moment, so would be happy to update when we've got something good.

[gianrubio]

@brancz so do you think operator will be responsible to manage blackbox-export as same way it take care of alertmanager?

What about a statefulset implementation of blackbox exporter?

[brancz]

The blackbox exporter doesn't have any state of its own, so a StatefulSet would be exaggerated, but some kind of abstraction would make sense, however, currently I am thinking of mainly abstracting the way the blackbox exporter is configured, rather than how it is run. I believe that the integration will look something like the integration of the Alertmanager in the Prometheus object, where we loosely reference a blackbox exporter running behind a specific Service, and then an abstracted form of blackbox probing configuration in a ServiceMonitor. This is all work in progress thinking, any input is appreciated.

[gianrubio]

@brancz I wrote a draft to manage blackbox exporter with prometheus operator as a tpr. Suggestions are welcomed

https://github.com/gianrubio/prometheus-operator/tree/blackbox-exporter/Documentation/proposals/blackbox-exporter#generated-prometheus-config

[galexrt]

@gianrubio Nice draft, but exactly do you mean with the question 1, "the http protocol (http x http) on the service spec"? Should it mean http and https?

[gianrubio]

@galexrt it was a typo, fixed now!

[galexrt]

@gianrubio How about changing the externalName to externalNames in the service and making it a list of ExternalName:

type ExternalName struct{
    name     string
    probe     string
    address string
}

For http probe the address would contain http:// or https:// and for others just the address without "protocol".

Edit: Just saw that the externalName is from Kubernetes service. How about making this into a ThirdPartyResource named ExternalService?

[brancz]

@gianrubio thanks a lot for your effort. I was two weeks on vacation, and am catching up on everything as quickly as I can. I have some comments regarding the proposal.

• I don't think the BlackboxExporter resources is necessary. As far as I can tell everything described by it is perfectly possible with a normal Deployment and a ConfigMap. I don't think users actually win anything as the same things will still have to be specified.

• The critical part is the integration that makes the Operator generate the right config, and tell it where to find the BlackboxExporter (I would think this is a namespace/service combination similar to the Alertmanager configuration). The configuration source should probably be the ServiceMonitor.

[ptagr]

@brancz were you able to add documentation on custom configuration as a workaround for this or can you point me to one? I am wanting to use blackbox-exporter using the operator to monitor our internal k8s services

[brancz]

@Punitag I haven't found the time, but the tl;dr is that you skip the serviceMonitorSelector and provide your own Secret named prometheus-<name-of-your-prometheus-object>, the two caveats are that you need to manually ensure the rule files and secrets that are mounted into the container if you need them. There have been issues around this in the past so if you can get around to writing this doc, that would be amazing! 🙂

[sebglon]

@brancz
I have found a good exemple of config for blackbox / kubernetes
prometheus/blackbox_exporter#113

Bit is it possible to add custom job config for specifi service monitor?
This can help all new exporter to inject there job definition.
Or add possibility to append relabel_config ???

[brancz]

For the time being I would recommend to use a custom prometheus config to run the blackbox exporter. We'll eventually have to figure out a meaningful abstraction to use it, but we're not there yet.

[sebglon]

Yes but is it possible to use custom promethues config with prometheus-operator?
This is a very bad lock for my project.
Can i help you

[brancz]

Yes we have a documentation page on this: https://github.com/coreos/prometheus-operator/blob/master/Documentation/custom-configuration.md

[sebglon]

Have you an example o using it?

[brancz]

I'm not sure I understand the question. The Prometheus object just must not have a serviceMonitorSelector field and you have to provide a Secret named prometheus-<name> that has a prometheus.yaml. The config is just like any other Prometheus configuration file.

[adamdecaf]

I struggled figuring out how to get this working too. I ended up with a Deployment (and Service) exposing blackbox_exporter and then configuring prometheus to scrape via blackbox_exporter's /probe endpoint.

I'd like to help out with some docs around this as it wasn't very clear.

blackbox_exporter Deployment

{
    "apiVersion": "apps/v1beta1",
    "kind": "Deployment",
    "metadata": {
        "name": "blackbox-exporter",
        "namespace": "legacy"
    },
    "spec": {
        "replicas": 1,
        "selector": {
            "matchLabels": {
                "app": "blackbox-exporter"
            }
        },
        "template": {
            "metadata": {
                "labels": {
                    "app": "blackbox-exporter"
                }
            },
            "spec": {
                "containers": [
                    {
                        "image": "registry/blackbox-exporter:v0.11.0",
                        "name": "blackbox-exporter",
                        "ports": [
                            {
                                "containerPort": 9115,
                                "name": "metrics"
                            }
                        ]
                    }
                ]
            }
        }
    }
}

blackbox_exporter Service

{
    "apiVersion": "v1",
    "kind": "Service",
    "metadata": {
        "labels": {
            "app": "blackbox-exporter"
        },
        "name": "blackbox-exporter",
        "namespace": "legacy"
    },
    "spec": {
        "ports": [
            {
                "name": "http-metrics",
                "port": 9115,
                "protocol": "TCP",
                "targetPort": "metrics"
            }
        ],
        "selector": {
            "app": "blackbox-exporter"
        }
    }
}

{
    "apiVersion": "monitoring.coreos.com/v1",
    "kind": "ServiceMonitor",
    "metadata": {
        "labels": {
            "k8s-app": "blackbox-exporter"
        },
        "name": "blackbox-exporter",
        "namespace": "tectonic-system"
    },
    "spec": {
        "endpoints": [
            {
                "interval": "60s",
                "port": "http-metrics"
            },
            {
                "interval": "60s",
                "params": {
                    "module": [
                        "http_2xx"
                    ],
                    "target": [
                        "10.10.1.120:9200"
                    ]
                },
                "path": "/probe",
                "targetPort": 9115
            }
        ],
        "namespaceSelector": {
            "matchNames": [
                "legacy"
            ]
        },
        "selector": {
            "app": "blackbox-exporter"
        }
    }
}

[stale]

This issue has been automatically marked as stale because it has not had any activity in last 60d. Thank you for your contributions.

[stale]

This issue has been automatically marked as stale because it has not had any activity in last 60d. Thank you for your contributions.

[brancz]

There is still work to be done to have a first class support for blackbox probing.

[stale]

This issue has been automatically marked as stale because it has not had any activity in last 60d. Thank you for your contributions.

[brancz]

#2832 is working on this API. Hopefully soon coming to your prometheus-operator deployments! :)

[stale]

This issue has been automatically marked as stale because it has not had any activity in last 60d. Thank you for your contributions.

[Sallyan]

Hi @iroller
I am trying to monitor kubernetes services using blackbox-exporter. But didn't make it to work.
i have been blocked for week, it would be great if you can give some help.
Below are my settings:
blackbox config:

    modules:
      http_2xx:
        prober: http
        http:
          method: GET
          preferred_ip_protocol: "ip4"
          valid_status_codes: [200]
      http_post_2xx:
        prober: http
        http:
          method: POST
      http_kubernetes_service:
        prober: http
        timeout: 5s
        http:
          headers:
            Accept: "*/*"
            Accept-Language: "en-US"
          tls_config:
            ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
          bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
          preferred_ip_protocol: "ip4"

Prometheus additional-scrape-configs

- job_name: blackbox-exporter-kubernetes-services
  tls_config:
    ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  metrics_path: /probe
  params:
    module: [http_2xx]
  kubernetes_sd_configs:
  - role: service
  relabel_configs:
  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_probed]
    action: keep
    regex: true
  - source_labels: [__address__]
    target_label: __param_target
  - target_label: __address__
    replacement: monitoring-blackbox-exporter.kyma-system.svc.cluster.local:9115
  - source_labels: [__param_target]
    target_label: instance
  - action: labelmap
    regex: __meta_kubernetes_service_label_(.+)
  - source_labels: [__meta_kubernetes_namespace]
    target_label: kubernetes_namespace
  - source_labels: [__meta_kubernetes_service_name]
    target_label: kubernetes_name

Error In Prometheus Targets
Get http://monitoring-blackbox-exporter.kyma-system.svc.cluster.local:9115/probe?module=http_2xx&target=kiali.my-ns.svc%3A20001: read tcp {ip}:40780->{ip}:9115: read: connection reset by peer

[stale]

This issue has been automatically marked as stale because it has not had any activity in last 60d. Thank you for your contributions.

[brancz]

#2832 is progressing nicely :)

[lilic]

Since we merged #2832 we can close this issue for now. If there is anything missing for this feature or bugs feel free to open a new issue. 🎉

[mossad-zika]

For future googlers:

helm install --generate-name --namespace monitoring -f custom-values.yaml prometheus-community/prometheus-blackbox-exporter

custom.values.yaml

  modules:
    tcp_connect:
      prober: tcp
      timeout: 5s

serviceMonitor:
  enabled: true
  targets:
      - name: clamav
        url: clamav.foo:3310
        labels:
          app: blackbox
          release: bar
        interval: 15s
        scrapeTimeout: 15s
        module: tcp_connect
        additionalMetricsRelabels: {}

[smeeklai]

To make the above solution work, you also need to configure prometheus as written in the note here.

set prometheus.prometheusSpec.podMonitorSelectorNilUsesHelmValues and prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues to false.

Another solution is to use Probe CRD. Here is an example. Note for the Probe CRD:

Probe, which declaratively specifies how groups of ingresses or static targets should be monitored. The Operator automatically generates Prometheus scrape configuration based on the definition

[mossad-zika]

@smeeklai looks like you moved it here

[jcdauchy-moodys]

Here now Blackbox IngressProbe