Traefik upgrade failed

[dominch]

Environmental Info:
K3s Version: v1.25.4+k3s1
Node(s) CPU architecture, OS, and Version: x86 (64bit)
Cluster Configuration: 6 nodes 1 master
Describe the bug:

I found failing pod helm-install-traefik-h5gjd with errors in log:

Error: UPGRADE FAILED: rendered manifests contain a resource that already exists. 
Unable to continue with update: IngressClass "traefik" in namespace "" exists and cannot be imported into the current release: invalid ownership metadata;
label validation error: missing key "app.kubernetes.io/managed-by": must be set to "Helm";
annotation validation error: missing key "meta.helm.sh/release-name": must be set to "traefik"; 
annotation validation error: missing key "meta.helm.sh/release-namespace": must be set to "kube-system"

It's restarting and fails consequently, everything regard k8s and traefik works and I have no problems with cluster, just want to solve this issue.

Steps To Reproduce:
Unknown, probably something after one of updates (last one from 1.24), everything work for me.

Expected behavior:
It should run with no issues.

Actual behavior:
Pod is restarting trying to do some upgrade to traefik.

Additional context / logs:
Full log of pod:

if [[ ${KUBERNETES_SERVICE_HOST} =~ .*:.* ]]; then
	echo "KUBERNETES_SERVICE_HOST is using IPv6"
	CHART="${CHART//%\{KUBERNETES_API\}%/[${KUBERNETES_SERVICE_HOST}]:${KUBERNETES_SERVICE_PORT}}"
else
	CHART="${CHART//%\{KUBERNETES_API\}%/${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT}}"
fi

set +v -x
+ [[ '' != \t\r\u\e ]]
+ export HELM_HOST=127.0.0.1:44134
+ HELM_HOST=127.0.0.1:44134
+ tiller --listen=127.0.0.1:44134 --storage=secret
+ helm_v2 init --skip-refresh --client-only --stable-repo-url https://charts.helm.sh/stable/
[main] 2023/02/01 09:44:25 Starting Tiller v2.17.0 (tls=false)
[main] 2023/02/01 09:44:25 GRPC listening on 127.0.0.1:44134
[main] 2023/02/01 09:44:25 Probes listening on :44135
[main] 2023/02/01 09:44:25 Storage driver is Secret
[main] 2023/02/01 09:44:25 Max history per release is 0
Creating /home/klipper-helm/.helm 
Creating /home/klipper-helm/.helm/repository 
Creating /home/klipper-helm/.helm/repository/cache 
Creating /home/klipper-helm/.helm/repository/local 
Creating /home/klipper-helm/.helm/plugins 
Creating /home/klipper-helm/.helm/starters 
Creating /home/klipper-helm/.helm/cache/archive 
Creating /home/klipper-helm/.helm/repository/repositories.yaml 
Adding stable repo with URL: https://charts.helm.sh/stable/ 
Adding local repo with URL: http://127.0.0.1:8879/charts 
$HELM_HOME has been configured at /home/klipper-helm/.helm.
Not installing Tiller due to 'client-only' flag having been set
++ timeout -s KILL 30 helm_v2 ls --all '^traefik$' --output json
++ jq -r '.Releases | length'
[storage] 2023/02/01 09:44:25 listing all releases with filter
+ V2_CHART_EXISTS=
+ [[ '' == \1 ]]
+ [[ '' == \v\2 ]]
+ [[ -f /config/ca-file.pem ]]
+ [[ -n '' ]]
+ shopt -s nullglob
+ helm_content_decode
+ set -e
+ ENC_CHART_PATH=/chart/traefik.tgz.base64
+ CHART_PATH=/tmp/traefik.tgz
+ [[ ! -f /chart/traefik.tgz.base64 ]]
+ return
+ [[ install != \d\e\l\e\t\e ]]
+ helm_repo_init
+ grep -q -e 'https\?://'
+ echo 'chart path is a url, skipping repo update'
chart path is a url, skipping repo update
+ helm_v3 repo remove stable
Error: no repositories configured
+ true
+ return
+ helm_update install --set-string global.systemDefaultRegistry=
+ [[ helm_v3 == \h\e\l\m\_\v\3 ]]
++ helm_v3 ls --all -f '^traefik$' --namespace kube-system --output json
++ jq -r '"\(.[0].app_version),\(.[0].status)"'
++ tr '[:upper:]' '[:lower:]'
Already installed traefik
+ LINE=2.6.2,deployed
+ IFS=,
+ read -r INSTALLED_VERSION STATUS _
+ VALUES=
+ for VALUES_FILE in /config/*.yaml
+ VALUES=' --values /config/values-01_HelmChart.yaml'
+ for VALUES_FILE in /config/*.yaml
+ VALUES=' --values /config/values-01_HelmChart.yaml --values /config/values-10_HelmChartConfig.yaml'
+ [[ install = \d\e\l\e\t\e ]]
+ [[ 2.6.2 =~ ^(|null)$ ]]
+ [[ deployed =~ ^(pending-install|pending-upgrade|pending-rollback)$ ]]
+ [[ deployed == \d\e\p\l\o\y\e\d ]]
+ echo 'Already installed traefik'
+ [[ helm_v3 == \h\e\l\m\_\v\3 ]]
+ helm_v3 mapkubeapis traefik --namespace kube-system
2023/02/01 09:44:26 Release 'traefik' will be checked for deprecated or removed Kubernetes APIs and will be updated if necessary to supported API versions.
2023/02/01 09:44:26 Get release 'traefik' latest version.
2023/02/01 09:44:26 Check release 'traefik' for deprecated or removed APIs...
2023/02/01 09:44:26 Finished checking release 'traefik' for deprecated or removed APIs.
2023/02/01 09:44:26 Release 'traefik' has no deprecated or removed APIs.
2023/02/01 09:44:26 Map of release 'traefik' deprecated or removed APIs to supported versions, completed successfully.
+ echo 'Upgrading helm_v3 chart'
+ echo 'Upgrading traefik'
Upgrading traefik
+ shift 1
+ helm_v3 upgrade --set-string global.systemDefaultRegistry= traefik https://10.43.0.1:443/static/charts/traefik-19.0.400.tgz --values /config/values-01_HelmChart.yaml --values /config/values-10_HelmChartConfig.yaml
Error: UPGRADE FAILED: rendered manifests contain a resource that already exists. Unable to continue with update: IngressClass "traefik" in namespace "" exists and cannot be imported into the current release: invalid ownership metadata; label validation error: missing key "app.kubernetes.io/managed-by": must be set to "Helm"; annotation validation error: missing key "meta.helm.sh/release-name": must be set to "traefik"; annotation validation error: missing key "meta.helm.sh/release-namespace": must be set to "kube-system"

[brandond]

Error: UPGRADE FAILED: rendered manifests contain a resource that already exists. Unable to continue with update: IngressClass "traefik" in namespace "" exists and cannot be imported into the current release: invalid ownership metadata; label validation error: missing key "app.kubernetes.io/managed-by": must be set to "Helm"; annotation validation error: missing key "meta.helm.sh/release-name": must be set to "traefik"; annotation validation error: missing key "meta.helm.sh/release-namespace": must be set to "kube-system"

Did you manually create the IngressClass outside the chart, before upgrading Traefik? The error indicates that it exists but is not managed by Helm. You can either delete that resource and let the chart create it, or add the annotations that it lists needing.

Just out of curiosity, what version of K3s are you upgrading from?

[dominch]

Sorry for late answer, just missed this one :/

I'm using k3s from about 1.19 and doing some upgrades from time to time,
probably about 1.21 I created that ingress class and deployed cert manager to run services on cluster.
last upgrade is from 1.24 to 1.25

ok, trying to fix that issue now

[dominch]

After adding those labels/annotation traefik was upgraded and seems that most of it is working,
Some ingress gives now 403 error because of whitelist filter and probably internalTrafficPolicy has changed