Panic with ServerSideDiff=true (but without IncludeMutationWebhook=true) #18020
Comments
|
I've run into the exact same situation today {
"Version": "v2.10.8+37b1cf5",
"BuildDate": "2024-04-26T13:20:04Z",
"GitCommit": "37b1cf5306f9c245f188c4c0566c23a0f80cdc65",
"GitTreeState": "clean",
"GoVersion": "go1.21.3",
"Compiler": "gc",
"Platform": "linux/amd64",
"KustomizeVersion": "v5.2.1 2023-10-19T20:13:51Z",
"HelmVersion": "v3.14.3+gf03cc04",
"KubectlVersion": "v0.26.11",
"JsonnetVersion": "v0.20.0"
}Set Recovered from panic: runtime error: invalid memory address or nil pointer dereference
goroutine 206 [running]:
runtime/debug.Stack()
/usr/local/go/src/runtime/debug/stack.go:24 +0x5e
github.com/argoproj/argo-cd/v2/controller.(*ApplicationController).processAppRefreshQueueItem.func1()n /go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:1470 +0x54
panic({0x3658d60?, 0x7189d60?})
/usr/local/go/src/runtime/panic.go:920 +0x270
k8s.io/apimachinery/pkg/util/managedfields.(*GvkParser).Type(...)
/go/pkg/mod/k8s.io/apimachinery@v0.26.11/pkg/util/managedfields/gvkparser.go:43
githu
b.com/argoproj/gitops-engine/pkg/diff.removeWebhookMutation(0xc01c6acd70, 0xc01b7ae850, 0x0, {0x50b4020?, 0x3d7cb48?})
/go/pkg/mod/github.com/argoproj/gitops-engine@v0.7.1-0.20240122213038-792124280fcc/pkg/diff/diff.go:210 +0xfd
github.com/argoproj/gitops-engine/pkg/diff.serverSideDiff(0xc01b7ae7f0, 0xc01b7ae85
0, {0xc006fc18a8, 0x8, 0x8})
/go/pkg/mod/github.com/argoproj/gitops-engine@v0.7.1-0.20240122213038-792124280fcc/pkg/diff/diff.go:176 +0x2bd
github.com/argoproj/gitops-engine/pkg/diff.ServerSideDiff(0xc01b7ae850?, 0xc003a718a8?, {0xc003a718a8?, 0x8?, 0xc0075e0d70?})
/go/pkg/mod/github.com/argoproj/gitops-engi
ne@v0.7.1-0.20240122213038-792124280fcc/pkg/diff/diff.go:138 +0x2a
github.com/argoproj/gitops-engine/pkg/diff.Diff(0xc01b7ae178, 0xc01b7ae180, {0xc003a718a8, 0x8, 0x8})
/go/pkg/mod/github.com/argoproj/gitops-engine@v0.7.1-0.20240122213038-792124280fcc/pkg/diff/diff.go:88 +0x1d6
github.com/argoproj/gitops-engin
e/pkg/diff.DiffArray({0xc00dbf8600, 0x29, 0x7fb710832a68?}, {0xc00dbf8400, 0x29?, 0xc01a3ce4e0?}, {0xc006fc18a8, 0x8, 0x8})
/go/pkg/mod/github.com/argoproj/gitops-engine@v0.7.1-0.20240122213038-792124280fcc/pkg/diff/diff.go:814 +0x132
github.com/argoproj/argo-cd/v2/util/argo/diff.StateDiffs({0xc011890580?, 0xc0
22c63460?, 0x1e?}, {0xc00dbf8200?, 0xc00c588380?, 0xe?}, {0x50c90c0, 0xc01b343680?})
/go/src/github.com/argoproj/argo-cd/util/argo/diff/diff.go:318 +0x74b
github.com/argoproj/argo-cd/v2/controller.(*appStateManager).CompareAppState(0xc00075e8c0, 0xc00ca36800, 0xc012bf2240, {0xc00c2ca500, 0x5, 0x8}, {0xc019ec400
0?, 0x5, 0x8}, 0x1, ...)
/go/src/github.com/argoproj/argo-cd/controller/state.go:651 +0x3e1f
github.com/argoproj/argo-cd/v2/controller.(*ApplicationController).processAppRefreshQueueItem(0xc0007da700)
/go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:1581 +0x1223
github.com/argoproj/argo-cd/v2
/controller.(*ApplicationController).Run.func3()
/go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:819 +0x25
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
/go/pkg/mod/k8s.io/apimachinery@v0.26.11/pkg/util/wait/wait.go:157 +0x33
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x0?,
{0x5071dc0, 0xc000dc6240}, 0x1, 0xc000082d20)
/go/pkg/mod/k8s.io/apimachinery@v0.26.11/pkg/util/wait/wait.go:158 +0xaf
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x0?, 0x3b9aca00, 0x0, 0x0?, 0x0?)
/go/pkg/mod/k8s.io/apimachinery@v0.26.11/pkg/util/wait/wait.go:135 +0x7f
k8s.io/apimachinery/pkg/util/wait.
Until(0x0?, 0x0?, 0x0?)
/go/pkg/mod/k8s.io/apimachinery@v0.26.11/pkg/util/wait/wait.go:92 +0x1e
created by github.com/argoproj/argo-cd/v2/controller.(*ApplicationController).Run in goroutine 99
/go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:818 +0x7ceAdd |
jannfis
added
component:core
|
@jackmaninov I just tested the kube-prometheus-stack helm chart applying the following application in my local kubernetes cluster: I wasn't able to reproduce the error. The application is synced as expected and no errors are returned in the UI or in the logs. I tested with Argo CD master build. Will run another test with 2.10.8 and confirm. Can you please validate with the application above? It would be great if you could provide a minimal |
@sherif-fanous not sure if this is just a typo in your message but the correct annotation is: Please confirm. |
Yes, that application causes the panic as well. The only modification I made was in argocd's namespace (not the destination namespace), which is instead something like |
Sorry that was a typo. I used the correct annotation |
|
@jackmaninov where are you testing this? Is this running in a local Kubernetes cluster? Which Kubernetes version are you running? |
This is in my (remote) development environment, running under Talos Linux v1.7.0, k8s version v1.30.0. Might be a bit too bleeding edge, but since @sherif-fanous is seeing this as well maybe it's not so k8s version specific. |
|
After a bit more digging, I find that there's a failure to call a webhook repeatedly while the application is in the bad state (paniced and stuck refreshing), so the panic could be in reaction to that: From my kube-apiserver log, this repeats rapidly:
I'll try to track down what's causing the bad certificate and see if it improves things. |
|
@sherif-fanous can you please confirm in which environment you are running your tests and what is the Kubernetes server version? |
|
@leoluz I'm using Talos Linux 1.7 running Kubernetes 1.30.0. This is a home lab cluster. |
|
Ok, small world... |
|
And, yes, I'm (now) aware argo-cd is only tested up to k8s v1.28 as per: https://argo-cd.readthedocs.io/en/stable/operator-manual/tested-kubernetes-versions/ |
|
I was having a TLS error (invalid cipher suite specified) during deployment and I had to apply this patch (#17569) on top of Argo CD 2.10.9 to make it work. I tested the patched version against Kubernetes 1.30 and wasn't able to reproduce the issue. I applied the prometheus-stack helm and it is working as expected. @jackmaninov @sherif-fanous Can you please try to deploy Argo CD built from |
|
I got a similar panic with The appset used: apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: kube-prometheus-stack
namespace: argocd
spec:
goTemplate: true
generators:
- list:
elements:
- name: blake
chartRevision: 55.1.0
valuesRevision: main
template:
metadata:
name: '{{.name}}-kube-prometheus-stack'
spec:
project: default
sources:
- chart: kube-prometheus-stack
repoURL: https://prometheus-community.github.io/helm-charts
targetRevision: '{{.chartRevision}}'
helm:
releaseName: kube-prometheus-stack
valueFiles:
- $values/values.yaml
- repoURL: https://github.com/blakepettersson/argocd-kube-prometheus-stack
targetRevision: '{{.valuesRevision}}'
ref: values
destination:
namespace: monitoring
name: '{{.name}}'
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
- ServerSideApply=truekubectl (I know I need to bump my k3s, it's way too old 😄) kubectl version
Client Version: v1.29.4
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.23.5+k3s1
WARNING: version difference between client (1.29) and server (1.23) exceeds the supported minor version skew of +/-1Working Argo CD version: argocd version
argocd-server: v2.9.12+487dce5.dirty
BuildDate: 2024-04-19T03:54:04Z
GitCommit: 487dce58bfa228a4f8bc22c626d1920f1617c066
GitTreeState: dirty
GoVersion: go1.21.3
Compiler: gc
Platform: linux/amd64
Kustomize Version: v5.2.1 2023-10-19T20:13:51Z
Helm Version: v3.13.2+g2a2fb3b
Kubectl Version: v0.24.17Non-working Argo CD version: argocd-server: v2.10.7+96c0c41.dirty
BuildDate: 2024-04-19T03:53:43Z
GitCommit: 96c0c41b8948b0738567bd5c748a3de0cec5d19e
GitTreeState: dirty
GoVersion: go1.21.3
Compiler: gc
Platform: linux/amd64
Kustomize Version: v5.2.1 2023-10-19T20:13:51Z
Helm Version: v3.14.3+gf03cc04
Kubectl Version: v0.26.11
Jsonnet Version: v0.20.0 |
How do we do that? Set image tag to |
I managed this today and running I did however try deploying the test application in a freshly set-up local Talos cluster, and the issue did not present itself using argo-cd EDIT: @sherif-fanous: deploying master involves pulling the git repo, running |
|
@blakepettersson Can you please take a look at argoproj/gitops-engine#574 |
|
@leoluz Can this issue please be re-opened? I built master and still getting the panic Recovered from panic: runtime error: invalid memory address or nil pointer dereference
goroutine 236 [running]:
runtime/debug.Stack()
/usr/local/go/src/runtime/debug/stack.go:24 +0x5e
github.com/argoproj/argo-cd/v2/controller.(*ApplicationController).processAppRefreshQueueItem.func1()
/go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:1485 +0x54
panic({0x37ee8c0?, 0x76c5910?})
/usr/local/go/src/runtime/panic.go:770 +0x132
k8s.io/apimachinery/pkg/util/managedfields.(*GvkParser).Type(...)
/go/pkg/mod/k8s.io/apimachinery@v0.29.2/pkg/util/managedfields/gvkparser.go:43
github.com/argoproj/gitops-engine/pkg/diff.removeWebhookMutation(0xc00c5b6530, 0xc00c5b6250, 0x0, {0x552c700?, 0x3f1dff0?})
/go/pkg/mod/github.com/argoproj/gitops-engine@v0.7.1-0.20240514190100-8a3ce6d85caa/pkg/diff/diff.go:210 +0xfd
github.com/argoproj/gitops-engine/pkg/diff.serverSideDiff(0xc00c5b61e0, 0xc00c5b6250, {0xc00dc5f898, 0x8, 0x8})
/go/pkg/mod/github.com/argoproj/gitops-engine@v0.7.1-0.20240514190100-8a3ce6d85caa/pkg/diff/diff.go:176 +0x2bd
github.com/argoproj/gitops-engine/pkg/diff.ServerSideDiff(0xc00c5b6250?, 0xc00dc5f898?, {0xc00dc5f898?, 0x8?, 0xc00de232c0?})
/go/pkg/mod/github.com/argoproj/gitops-engine@v0.7.1-0.20240514190100-8a3ce6d85caa/pkg/diff/diff.go:138 +0x2a
github.com/argoproj/gitops-engine/pkg/diff.Diff(0xc00cece000, 0xc00cece008, {0xc00dc5f898, 0x8, 0x8})
/go/pkg/mod/github.com/argoproj/gitops-engine@v0.7.1-0.20240514190100-8a3ce6d85caa/pkg/diff/diff.go:88 +0x1cf
github.com/argoproj/gitops-engine/pkg/diff.DiffArray({0xc00bdcc800, 0x33, 0x7?}, {0xc00bdcc600, 0x33?, 0xc00a629f80?}, {0xc00dc5f898, 0x8, 0x8})
/go/pkg/mod/github.com/argoproj/gitops-engine@v0.7.1-0.20240514190100-8a3ce6d85caa/pkg/diff/diff.go:821 +0x132
github.com/argoproj/argo-cd/v2/util/argo/diff.StateDiffs({0xc004ab71e0?, 0x3c5ae00?, 0x1e?}, {0xc00f386800?, 0xc00c6fe7e0?, 0x7?}, {0x5542f20, 0xc00932f680})
/go/src/github.com/argoproj/argo-cd/util/argo/diff/diff.go:318 +0x74e
github.com/argoproj/argo-cd/v2/controller.(*appStateManager).CompareAppState(0xc0005f8e00, 0xc00737ec08, 0xc00a205448, {0xc00dc1db00, 0x4, 0x4}, {0xc00dc54e00, 0x4, 0x4}, 0x0, ...)
/go/src/github.com/argoproj/argo-cd/controller/state.go:684 +0x3d3f
github.com/argoproj/argo-cd/v2/controller.(*ApplicationController).processAppRefreshQueueItem(0xc000789880)
/go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:1595 +0x1183
github.com/argoproj/argo-cd/v2/controller.(*ApplicationController).Run.func3()
/go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:833 +0x25
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
/go/pkg/mod/k8s.io/apimachinery@v0.29.2/pkg/util/wait/backoff.go:226 +0x33
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000de4df0, {0x54e88e0, 0xc00119f1d0}, 0x1, 0xc000dd8de0)
/go/pkg/mod/k8s.io/apimachinery@v0.29.2/pkg/util/wait/backoff.go:227 +0xaf
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000de4df0, 0x3b9aca00, 0x0, 0x1, 0xc000dd8de0)
/go/pkg/mod/k8s.io/apimachinery@v0.29.2/pkg/util/wait/backoff.go:204 +0x7f
k8s.io/apimachinery/pkg/util/wait.Until(...)
/go/pkg/mod/k8s.io/apimachinery@v0.29.2/pkg/util/wait/backoff.go:161
created by github.com/argoproj/argo-cd/v2/controller.(*ApplicationController).Run in goroutine 125
/go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:832 +0x865I've not used Go in a long time but looking at the trace my understanding is that the panic is resulting from func (p *GvkParser) Type(gvk schema.GroupVersionKind) *typed.ParseableType {
typeName, ok := p.gvks[gvk]
if !ok {
return nil
}
t := p.parser.Type(typeName)
return &t
}This would indicate to me that either |
|
Also having this issue. Regarding config and versions shared in this thread, most things are the same but I'm using k3s 1.30. Running the newest 2.9.15 version works but anything above that (2.10+) gives me this panic. |
Checklist:
argocd version.Describe the bug
While attempting to enable ServerSideDiff in order to get the green checkmark on my deployment of kube-prometheus-stack:57.1.1, I encountered a panic which I've pasted below.
After enabling ServerSideDiff=true via the web interface, my application gets stuck refreshing, and manually attempting to sync gives the panic notification, though my logs indicate it occurs almost immediately after enabling the annotation.
I noticed the reference to removeWebhookMutation in the stack trace, turned on
IncludeMutationWebhook=trueas well and everything worked as expected.I was able to reproduce this on my install even on very simple applications. I'm sure more information about my config will be relevant, but I would need some guidance on what to contribute.
To Reproduce
argocd.argoproj.io/compare-options: ServerSideDiff=true. I did this by going to Details -> Edit, then added a new line under annotationsExpected behavior
Perhaps an error of some sort, but not a panic.
Screenshots
Version
Logs
Please excuse them being a bit munged, they came out of my logs as a single line.
The text was updated successfully, but these errors were encountered: