Skip to content

[Helm] Allow Service creation and custom annotations for Prometheus scraping (non-SQS mode) #1266

@lvg-dexters

Description

@lvg-dexters

Summary

In the current Helm chart (v0.27.6, app v1.25.6), the Service template (config/helm/aws-node-termination-handler/templates/service.yaml) has two limitations for users running NTH in IMDS/DaemonSet mode with enablePrometheusServer=true who want their metrics endpoint discoverable by Kubernetes-native Prometheus / Grafana Alloy setups:

  1. No Service is created in IMDS mode, even with enablePrometheusServer=true. The template condition requires both enableSqsTerminationDraining AND enablePrometheusServer to be true.
  2. Even when created (SQS mode), the Service has no annotations metadata block and no values hook to add them.

podMonitor.create handles this for Prometheus Operator users, but operators that rely on endpointslice / service-annotation discovery (vanilla Prometheus, Grafana Alloy, grafana/k8s-monitoring) have no chart-native path in IMDS mode.

Proposal (strictly additive, backward-compatible)

Two small additions, both opt-in and default-preserving:

  1. Add service.create: false (boolean, default false). When set to true in IMDS mode, creates a headless Service (clusterIP: None) selecting DaemonSet pods. In SQS mode, it is redundant (Service already auto-created).
  2. Add service.annotations: {} (default empty). When set, applied to the Service metadata.

Behavior matrix (proof that no existing user sees change)

enableSqsTerminationDraining enablePrometheusServer service.create Service created? vs today
any false any No same
true true false Yes (ClusterIP, deployment selector) same
true true true Yes (ClusterIP, deployment selector) same (opt-in no-op in SQS mode)
false true false No same
false true true Yes (headless, daemonset selector) new (opt-in only)
  • Existing SQS users: zero change.
  • Existing IMDS users who do not set service.create: zero change.
  • Only IMDS users who explicitly set service.create=true see the new Service.

Implementation sketch

Template condition becomes:

{{- if and .Values.enablePrometheusServer (or .Values.enableSqsTerminationDraining .Values.service.create) -}}

Labels and selector branch on mode:

  • SQS mode: existing labelsDeployment / selectorLabelsDeployment (unchanged).
  • IMDS mode: labelsDaemonset / selectorLabelsDaemonset (both helpers already exist in _helpers.tpl, verified).

Service spec in IMDS mode adds clusterIP: None since the natural shape for DaemonSet metric discovery is headless (one endpoint per node pod, scraped individually by Prometheus / Alloy). SQS mode keeps the existing ClusterIP.

The service.annotations block threads through in either mode.

Search

Checked existing issues and PRs for service.annotations, serviceAnnotations, and related terms: zero matches.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions