sts recreate job v2.10.3

[KaloyanHr]

Hello,

I upgraded this week my tempo-distributed instances to v2.10.3 using helm chart v2.8.0. When I tried testing the sts recreate job for expanding PVC size of ingesters and metrics-generators I ran in to the following issues:

patch-placeholder Error from server (Forbidden): persistentvolumeclaims "placeholder-ingester-7" is forbidden: User "system:serviceaccount:tempo:tempo-distributed-ingester-recreate" cannot get resource "persistentvolumeclaims" in API group "" in the namespace "tempo"

patch-placeholder-2 error: --type must be one of [json merge strategic[], not "'json'"

Also errors related to the v1 apiGroup.

After checking the _helpers.tpl, I had to make the following changes:

Add get verb and change the apiGroup to core ("") instead of v1

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: {{ $newStatefulSet.metadata.name }}-recreate
  namespace: {{ $newStatefulSet.metadata.namespace }}
  labels:
    {{- $newStatefulSet.metadata.labels | toYaml | nindent 4 }}
    app.kubernetes.io/component: statefulset-recreate-job
  annotations:
    "helm.sh/hook": pre-upgrade
    "helm.sh/hook-weight": "-10"
    "helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded
rules:
  - apiGroups:
      - apps
    resources:
      - statefulsets
    resourceNames:
      - {{ $newStatefulSet.metadata.name }}
    verbs:
      - delete
  {{- if $templates }}
  - apiGroups:
      - ""
    resources:
      - persistentvolumeclaims
    resourceNames:
    {{- range $index := until (int $currentStatefulset.spec.replicas) }}
      {{- range $template := $templates | keys }}
      - {{ printf "%s-%s-%d" $template $newStatefulSet.metadata.name $index }}
      {{- end }}
    {{- end }}
    verbs:
      - get
      - patch
  {{- end }}

And update the command for the recreate job to:

apiVersion: batch/v1
kind: Job
metadata:
  name: {{ $newStatefulSet.metadata.name }}-recreate
  namespace: {{ $newStatefulSet.metadata.namespace }}
  labels:
    {{- $newStatefulSet.metadata.labels | toYaml | nindent 4 }}
    app.kubernetes.io/component: statefulset-recreate-job
  annotations:
    "helm.sh/hook": pre-upgrade
    "helm.sh/hook-weight": "-5"
    "helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded
spec:
  ttlSecondsAfterFinished: 300
  template:
    metadata:
      name: {{ $newStatefulSet.metadata.name }}-recreate
      labels:
        {{- $newStatefulSet.metadata.labels | toYaml | nindent 8 }}
    spec:
      serviceAccountName: {{ $newStatefulSet.metadata.name }}-recreate
      restartPolicy: OnFailure
      containers:
        - name: recreate
          image: {{ printf "%s:%s" (coalesce $.context.Values.kubectlImage.repository "registry.k8s.io") $.context.Values.kubectlImage.tag }}
          command:
            - kubectl
            - delete
            - statefulset
            - {{ $newStatefulSet.metadata.name }}
            - --cascade=orphan
        {{- range $index := until (int $currentStatefulset.spec.replicas) }}
          {{- range $template, $size := $templates }}
        - name: patch-pvc-{{ $template }}-{{ $index }}
          image: {{ printf "%s:%s" (coalesce $.context.Values.kubectlImage.repository "registry.k8s.io") $.context.Values.kubectlImage.tag }}
          command:
            - kubectl
            - patch
            - pvc
            - --namespace={{ $newStatefulSet.metadata.namespace }}
            - {{ printf "%s-%s-%d" $template $newStatefulSet.metadata.name $index }}
            - --type=json
            - -p
            - '[{"op":"replace","path":"/spec/resources/requests/storage","value":"{{ $size }}"}]'
          {{- end }}
        {{- end }}

I also updated the image value here since I'm pulling the image from my own artifactory and {{ printf "%s/kubectl:%s" (coalesce $.context.Values.global.image.registry "registry.k8s.io") $.context.Capabilities.KubeVersion.Version }} wasn't working well for my setup.

This was tested on k8s v1.31.