Is there a way to open ports for Prometheus to scrape?

[afilipchik]

No description provided.

[functicons]

Thanks for the proposal! I need to do some research first. If you have more detailed ideas, we can discuss here.

[functicons]

@afilipchik Does this work for you? I think you might need to build a custom image and specify related Flink properties in the CR.

I don't see any change needed in the operator.

[chethanuk]

custom image

@afilipchik Check https://github.com/ChethanUK/Flink_Docker/blob/master/Dockerfile to build a custom image

and then In job-manager-service, I add the following lines:

  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "9999"

[functicons]

@chethanuk Thanks for your image, but why are the annotations needed for JM service?

[afilipchik]

I thought the operator will need to open appropriate port (in the generated yaml), so the scraper can reach the container. Currently only flink related ports are exposed.

For now I did build a custom image and used prometheus push gateway, but having direct scrape sound more appropriate.

[functicons]

@afilipchik right, I think we can expose a prometheus port (9999 by default) on JM and TMs.

[functicons]

I realized the hard part is that, Prometheus needs to distinguish each TM, e.g.

 scrape_configs:
 - job_name: 'flink'
   static_configs:
   - targets: ['job-cluster:9999', 'taskmanager1:9999', 'taskmanager2:9999']

that makes it hard for Kubernetes, because we cannot simply create a TM service which is backed by all TMs.

The doc says:

In more dynamic scenarios we recommend using Prometheus’ service discovery support for different platforms such as Kubernetes, AWS EC2 and more.

I need to do more research on this.

[afilipchik]

I believe Prometheus can discover scrape targets (no need to hardcode them in the config). Target just needs to be accessible.

- job_name: 'kubernetes-service-endpoints'
   kubernetes_sd_configs:
    - role: endpoints

[elanv]

Prometheus also supports pod discovery. Using this method, prometheus queries the k8s api with pod meta information - labels and annotations - to get pod IPs and scrapes directly with those IPs.

Note: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#pod

To support this approach we need to map the exporter port to the pod and allow the pod's label and annotation to be set. In this case metric collection is possible without any services.

[elanv]

It seems this is resolved via #241, #243.