[Feature]: Allow specific annotation to be set on dgcm-exporter pod's daemonset

[MadJlzz]

Description

Add support to set extra annotations to dcgm-exporter only.
Solves #2271

Checklist

• No secrets, sensitive information, or unrelated changes
• Lint checks passing (make lint)
• Generated assets in-sync (make validate-generated-assets)
• Go mod artifacts in-sync (make validate-modules)
• Test cases are added for new code paths

Testing

Added multiple units tests for now:

• add annotations on dcgm-exporter when daemonsets.annotations is empty
• combine annotations of dcgm-exporter and daemonsets.annotations when both are set

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[MadJlzz]

Hello there maintainers 👋🏼

This is a naive approach to add extra annotations to dcgm-exporter daemonset as well as it's pod. I have nothing against reviewing the entire logic - this is my first time touching a Kubernetes operator.

[rahulait]

Thanks @MadJlzz. You'll also need to update helm chart templates to render newly added annotations field.
See here:

gpu-operator/deployments/gpu-operator/templates/clusterpolicy.yaml

Lines 541 to 585 in 8d9d357

	dcgmExporter:
	enabled: {{ .Values.dcgmExporter.enabled }}
	{{- if .Values.dcgmExporter.repository }}
	repository: {{ .Values.dcgmExporter.repository }}
	{{- end }}
	{{- if .Values.dcgmExporter.image }}
	image: {{ .Values.dcgmExporter.image }}
	{{- end }}
	{{- if .Values.dcgmExporter.version }}
	version: {{ .Values.dcgmExporter.version | quote }}
	{{- end }}
	{{- if .Values.dcgmExporter.imagePullPolicy }}
	imagePullPolicy: {{ .Values.dcgmExporter.imagePullPolicy }}
	{{- end }}
	{{- if .Values.dcgmExporter.imagePullSecrets }}
	imagePullSecrets: {{ toYaml .Values.dcgmExporter.imagePullSecrets | nindent 6 }}
	{{- end }}
	{{- if .Values.dcgmExporter.resources }}
	resources: {{ toYaml .Values.dcgmExporter.resources | nindent 6 }}
	{{- end }}
	{{- if .Values.dcgmExporter.env }}
	env: {{ toYaml .Values.dcgmExporter.env | nindent 6 }}
	{{- end }}
	{{- if .Values.dcgmExporter.args }}
	args: {{ toYaml .Values.dcgmExporter.args | nindent 6 }}
	{{- end }}
	{{- if and (.Values.dcgmExporter.config) (.Values.dcgmExporter.config.name) }}
	config:
	name: {{ .Values.dcgmExporter.config.name }}
	{{- end }}
	{{- if .Values.dcgmExporter.serviceMonitor }}
	serviceMonitor: {{ toYaml .Values.dcgmExporter.serviceMonitor | nindent 6 }}
	{{- end }}
	{{- if .Values.dcgmExporter.service }}
	service: {{ toYaml .Values.dcgmExporter.service | nindent 6 }}
	{{- end }}
	{{- if .Values.dcgmExporter.hostPID }}
	hostPID: {{ .Values.dcgmExporter.hostPID }}
	{{- end }}
	{{- if .Values.dcgmExporter.hostNetwork }}
	hostNetwork: {{ .Values.dcgmExporter.hostNetwork }}
	{{- end }}
	{{- if .Values.dcgmExporter.hpcJobMapping }}
	hpcJobMapping: {{ toYaml .Values.dcgmExporter.hpcJobMapping | nindent 6 }}
	{{- end }}

And then update dcgmExporter's values in values.yaml with an example

gpu-operator/deployments/gpu-operator/values.yaml

Lines 280 to 328 in 8d9d357

	dcgmExporter:
	enabled: true
	repository: nvcr.io/nvidia/k8s
	image: dcgm-exporter
	version: 4.5.1-4.8.0-distroless
	imagePullPolicy: IfNotPresent
	env: []
	resources: {}
	hostPID: false
	hostNetwork: false
	# HPC job mapping configuration for correlating GPU metrics with HPC workload manager jobs
	# This is used by HPC workload managers like Slurm to label GPU metrics with job IDs
	# hpcJobMapping:
	# enabled: true
	# directory: /var/lib/dcgm-exporter/job-mapping
	service:
	internalTrafficPolicy: Cluster
	serviceMonitor:
	enabled: false
	interval: 15s
	honorLabels: false
	additionalLabels: {}
	relabelings: []
	# - source_labels:
	# - __meta_kubernetes_pod_node_name
	# regex: (.*)
	# target_label: instance
	# replacement: $1
	# action: replace
	# DCGM Exporter configuration
	# This block is used to configure DCGM Exporter to emit a customized list of metrics.
	# Use "name" to either point to an existing ConfigMap or to create a new one with a
	# list of configurations (i.e with create=true).
	# When pointing to an existing ConfigMap, the ConfigMap must exist in the same namespace as the release.
	# The metrics are expected to be listed under a key called `dcgm-metrics.csv`.
	# Use "data" to build an integrated ConfigMap from a set of custom metrics as
	# part of the chart. An example of some custom metrics are shown below. Note that
	# the contents of "data" must be in CSV format and be valid DCGM Exporter metric configurations.
	# config:
	# name: custom-dcgm-exporter-metrics
	# create: true
	# data: |-
	# Format
	# If line starts with a '#' it is considered a comment
	# DCGM FIELD, Prometheus metric type, help message
	# Clocks
	# DCGM_FI_DEV_SM_CLOCK, gauge, SM clock frequency (in MHz).
	# DCGM_FI_DEV_MEM_CLOCK, gauge, Memory clock frequency (in MHz).

[rahulait]

/ok to test bef9213

[rahulait]

Thanks @MadJlzz. You can test these changes and see if the built image works fine for you.

Steps to verify:

1. Checkout this branch.
2. Create a custom values.yaml which includes dcgmExporter with annotations and also overwrite gpu-operator image with this values.yaml:

operator:
  repository: ghcr.io/nvidia
  image: gpu-operator
  version: bef9213a

dcgmExporter:
  annotations:
    <your annotations here>

3. Helm install

helm install gpu-operator ./deployments/gpu-operator -n gpu-operator -f values.yaml

[MadJlzz]

It looks like everything is working as expected.

[rahulait]

Overall, the PR looks good to me. I'll let other maintainers take a look as well and merge as required.

[rahulait]

/ok to test c2e0c1f

[tariq1890]

/ok to test 7255ce0

[tariq1890]

Thank you @MadJlzz !