[stable/concourse] Segregate Prometheus service from Web service

[cirocosta]

Hey,

With the addition of the Prometheus port to web-svc in #6338, an operator exposing the web service publicly using the service type LoadBalancer ends up exposing their metrics out there.

For instance:

web:
  service:
    type: LoadBalancer
    loadBalancerIP: 1.2.3.4
concourse:
  web:
    prometheus:
      enabled: true

produces:

# Source: concourse/templates/web-svc.yaml
apiVersion: v1
kind: Service
metadata:
  name: release-name-web
  labels:
    app: release-name-web
    chart: "concourse-3.6.2"
    release: "release-name"
    heritage: "Tiller"
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "9391"
spec:
  type: LoadBalancer
  loadBalancerIP: 1.2.3.4
  ports:
    - name: atc
      port: 8080
      targetPort: atc
    - name: tsa
      port: 2222
      targetPort: tsa
    - name: prometheus
      port: 9391
      targetPort: prometheus
  selector:
    app: release-name-web

It seems to me that to solve that while still supporting the use case described in the PR mentioned above, we could have a separate Prometheus service.

Maybe that is something that we could enable by default to be backward compatible, but leave that configurable under a separate field.

For instance:

web:
  service:
    type: LoadBalancer
    loadBalancerIP: 1.2.3.4
concourse:
  web:
    prometheus:
      enabled: true

prometheus:
  service:
    enabled: true
    # some additional properties ...

With such configuration, the following scenarios could exist:

• web service w/ Prometheus scraping the pods directly (from user-configured additional pod annotations);
• web service w/ Prometheus scraping the separate service (private); and
• web service w/ Prometheus scraping the separate service (public).

Wdyt?

Thanks!

---

cc @william-tran

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[cirocosta]

#11289 (comment) 🙌

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[stale]

This issue is being automatically closed due to inactivity.