CFP: Cilium ServiceMonitor objects gets created without 'relabelings.action' in the relabelings stanza

[kdandersen]

Cilium Feature Proposal

Don't know for sure if this a bug or a feature requset...

We have discovered the following issue when deploying Cilium in collaboration with ArgoCD as deployment engine for the Helm chart.

When enabling ServiceMonitor objects to be discovered via Prometheus Operator the ServiceMonitor object relabelling defininition gets created like this:

...
spec:
  selector:
    matchLabels:
      k8s-app: cilium
  namespaceSelector:
    matchNames:
    - kube-system
  endpoints:
  - port: metrics
    interval: "10s"
    honorLabels: true
    path: /metrics
    relabelings:
    - replacement: ${1}
      sourceLabels:
      - __meta_kubernetes_pod_node_name
      targetLabel: node
  targetLabels:
  - k8s-app

This gets modified by the Prometheus Operator admission webhook to the following format (adding action: replace):

...
spec:
  selector:
    matchLabels:
      k8s-app: cilium
  namespaceSelector:
    matchNames:
    - pl-cilium
  endpoints:
  - port: metrics
    interval: 10s
    honorLabels: true
    path: /metrics
    relabelings:
    - replacement: ${1}
      action: replace
      sourceLabels:
      - __meta_kubernetes_pod_node_name
      targetLabel: node
  targetLabels:
  - k8s-app

This again results in ArgoCD detecting drift and reconciling the ServiceMonitor back to its original spec. without the action: replace section. Resulting in an eternal OutOfSynch reconciliation loop.

For now we will try to solve this via ArgoCD ignoreDifferences configuration. But we would like/propose the Cilium ServiceMonitor objects to include the action: replace in a future release.

Kind regards.

[github-actions]

This issue has been automatically marked as stale because it has not
had recent activity. It will be closed if no further activity occurs.

[github-actions]

This issue has not seen any activity since it was marked stale.
Closing.

[David-VTUK]

If anyone is encountering this whilst using ArgoCD ApplicationSet objects, here's my example for adding this exclusion using templatePatch and IgnoreDifferences

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: bootstrap-applications
  namespace: argocd
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
  - git:
      repoURL: 'https://github.com/David-VTUK/turing-pi-automation.git'
      revision: HEAD
      directories:
      - path: 'argocd-apps/helm-charts/import-from-cluster-standup/*'
  template:
    metadata:
      name: '{{ .path.basename }}'
    spec:
      project: default
      source:
        repoURL: 'https://github.com/David-VTUK/turing-pi-automation.git'
        targetRevision: HEAD
        path: '{{ .path.path }}'
        helm:
          valueFiles:
            - values.yaml
      destination:
        server: 'https://kubernetes.default.svc'
        namespace: '{{ .path.basename }}'
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
          - CreateNamespace=true
  templatePatch: |
    {{ if eq .path.basename "cilium" }}
      spec:
        ignoreDifferences:
        - group: monitoring.coreos.com
          kind: ServiceMonitor
          name: ""
          jsonPointers:
            - /spec
        syncPolicy:
          syncOptions:
            - ServerSideApply=true # required to avoid the "annotations too long error"
            - CreateNamespace=true
    {{- end }}