Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

argocd-application-controller crashed: makeslice: cap out of range #4933

Closed
alexmt opened this issue Nov 30, 2020 · 6 comments · Fixed by #4965 or #7463
Closed

argocd-application-controller crashed: makeslice: cap out of range #4933

alexmt opened this issue Nov 30, 2020 · 6 comments · Fixed by #4965 or #7463
Assignees
@mayzhang2000
Labels
bug Something isn't working
Milestone
v2.2

Comments

@alexmt
Copy link
Collaborator

alexmt commented Nov 30, 2020 •
edited

Describe the bug

argocd-application-controller crashed: makeslice: cap out of range. Issue disappeared after restart.

To Reproduce

No clear steps to reproduce

Version

v1.8.0-rc1

Logs

Recovered from panic: runtime error: makeslice: cap out of range
goroutine 317 [running]:
runtime/debug.Stack(0xc093742d80, 0x1e9ba40, 0x24ba180)
    /usr/local/go/src/runtime/debug/stack.go:24 +0x9d
github.com/argoproj/argo-cd/controller.(*ApplicationController).processAppRefreshQueueItem.func1(0xc00025f7c0, 0x1def500, 0xc08b707170)
    /go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:991 +0x88
panic(0x1e9ba40, 0x24ba180)
    /usr/local/go/src/runtime/panic.go:975 +0x3e3
github.com/vmihailenco/bufpool.(*bufPool).Get(0x38b8060, 0x2007327, 0x7f212ee8e2a0)
    /go/pkg/mod/github.com/vmihailenco/bufpool@v0.1.11/buf_pool.go:45 +0x273
github.com/vmihailenco/bufpool.Get(...)
    /go/pkg/mod/github.com/vmihailenco/bufpool@v0.1.11/buf_pool.go:17
github.com/vmihailenco/bufpool.(*Buffer).grow(0xc09f0b0540, 0x7327, 0x1bb2b43)
    /go/pkg/mod/github.com/vmihailenco/bufpool@v0.1.11/buffer_ext.go:57 +0x172
github.com/vmihailenco/bufpool.(*Buffer).Write(0xc09f0b0540, 0xc0c5a88000, 0x7327, 0x7327, 0xc093743028, 0x1bb3f9e, 0xc095a3c0f0)
    /go/pkg/mod/github.com/vmihailenco/bufpool@v0.1.11/buffer.go:121 +0xdc
github.com/vmihailenco/msgpack/v5.(*Encoder).writeString(0xc095a3c0f0, 0xc0c5a88000, 0x7327, 0x0, 0x199)
    /go/pkg/mod/github.com/vmihailenco/msgpack/v5@v5.0.0-alpha.2/encode.go:245 +0x75
github.com/vmihailenco/msgpack/v5.(*Encoder).EncodeString(0xc095a3c0f0, 0xc0c5a88000, 0x7327, 0xc0c5a88000, 0x7327)
    /go/pkg/mod/github.com/vmihailenco/msgpack/v5@v5.0.0-alpha.2/encode_slice.go:70 +0x7e
github.com/vmihailenco/msgpack/v5.encodeStringValue(0xc095a3c0f0, 0x1def500, 0xc0c5429fb8, 0x198, 0x1, 0x1)
    /go/pkg/mod/github.com/vmihailenco/msgpack/v5@v5.0.0-alpha.2/encode_slice.go:12 +0x61
github.com/vmihailenco/msgpack/v5.(*field).EncodeValue(0xc0413fe800, 0xc095a3c0f0, 0x20afe80, 0xc0c5429f40, 0x199, 0x0, 0x0)
    /go/pkg/mod/github.com/vmihailenco/msgpack/v5@v5.0.0-alpha.2/types.go:105 +0xea
github.com/vmihailenco/msgpack/v5.encodeStructValue(0xc095a3c0f0, 0x20afe80, 0xc0c5429f40, 0x199, 0xc0c5429f40, 0x199)
    /go/pkg/mod/github.com/vmihailenco/msgpack/v5@v5.0.0-alpha.2/encode_map.go:154 +0x20f
github.com/vmihailenco/msgpack/v5.ptrEncoderFunc.func1(0xc095a3c0f0, 0x20a9a20, 0xc0e6cad138, 0x196, 0x20a9a20, 0x0)
    /go/pkg/mod/github.com/vmihailenco/msgpack/v5@v5.0.0-alpha.2/encode_value.go:123 +0xa8
github.com/vmihailenco/msgpack/v5.(*Encoder).EncodeValue(0xc095a3c0f0, 0x20a9a20, 0xc0e6cad138, 0x196, 0x20a9a20, 0xc0e6cad138)
    /go/pkg/mod/github.com/vmihailenco/msgpack/v5@v5.0.0-alpha.2/encode.go:217 +0x8c
github.com/vmihailenco/msgpack/v5.encodeArrayValue(0xc095a3c0f0, 0x1d8a520, 0xc018faa460, 0x97, 0x2291c88, 0x20e2201)
    /go/pkg/mod/github.com/vmihailenco/msgpack/v5@v5.0.0-alpha.2/encode_slice.go:126 +0xe2
github.com/vmihailenco/msgpack/v5.encodeSliceValue(0xc095a3c0f0, 0x1d8a520, 0xc018faa460, 0x97, 0x1d8a520, 0x20e2240)
    /go/pkg/mod/github.com/vmihailenco/msgpack/v5@v5.0.0-alpha.2/encode_slice.go:117 +0x78
github.com/vmihailenco/msgpack/v5.(*Encoder).EncodeValue(0xc095a3c0f0, 0x1d8a520, 0xc018faa460, 0x97, 0xc09f0b0540, 0x7f2133604168)
    /go/pkg/mod/github.com/vmihailenco/msgpack/v5@v5.0.0-alpha.2/encode.go:217 +0x8c
github.com/vmihailenco/msgpack/v5.(*Encoder).Encode(0xc095a3c0f0, 0x1d8a520, 0xc018faa460, 0x203027, 0x203027)
    /go/pkg/mod/github.com/vmihailenco/msgpack/v5@v5.0.0-alpha.2/encode.go:203 +0xe9
github.com/go-redis/cache/v8.(*Cache).Marshal(0xc0006442d0, 0x1d8a520, 0xc018faa460, 0x0, 0x0, 0x0, 0x0, 0x0)
    /go/pkg/mod/github.com/go-redis/cache/v8@v8.0.0-beta.11/cache.go:352 +0x16b
github.com/go-redis/cache/v8.(*Cache).set(0xc0006442d0, 0xc09ca48fa0, 0xc01b60b000, 0x36, 0xc0937434c0, 0x40cd28, 0x30, 0x1f79cc0)
    /go/pkg/mod/github.com/go-redis/cache/v8@v8.0.0-beta.11/cache.go:140 +0x64
github.com/go-redis/cache/v8.(*Cache).Set(...)
    /go/pkg/mod/github.com/go-redis/cache/v8@v8.0.0-beta.11/cache.go:130
github.com/argoproj/argo-cd/util/cache.(*redisCache).Set(0xc00000d7a0, 0xc0c73ea450, 0xc093743518, 0x2)
    /go/src/github.com/argoproj/argo-cd/util/cache/redis.go:33 +0xb0
github.com/argoproj/argo-cd/util/cache.(*Cache).SetItem(0xc000528ca0, 0xc05d82df40, 0x36, 0x1d8a520, 0xc018faa460, 0x34630b8a000, 0x0, 0xc0baa92b70, 0x30)
    /go/src/github.com/argoproj/argo-cd/util/cache/cache.go:89 +0x164
github.com/argoproj/argo-cd/util/cache/appstate.(*Cache).SetItem(...)
    /go/src/github.com/argoproj/argo-cd/util/cache/appstate/cache.go:51
github.com/argoproj/argo-cd/util/cache/appstate.(*Cache).SetAppManagedResources(0xc000528cb0, 0xc095f2a960, 0x1a, 0xc0e6cac900, 0x10a, 0x10a, 0x0, 0x0)
    /go/src/github.com/argoproj/argo-cd/util/cache/appstate/cache.go:64 +0x125
github.com/argoproj/argo-cd/controller.(*ApplicationController).setAppManagedResources(0xc00025f7c0, 0xc08f25c400, 0xc0b9d6d970, 0x9, 0xc093743c98, 0xc06555f000)
    /go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:280 +0x14a
github.com/argoproj/argo-cd/controller.(*ApplicationController).processAppRefreshQueueItem(0xc00025f7c0, 0x203001)
    /go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:1077 +0x8c3
github.com/argoproj/argo-cd/controller.(*ApplicationController).Run.func3()
    /go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:475 +0x36
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc000925d40)
    /go/pkg/mod/k8s.io/apimachinery@v0.19.2/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000925d40, 0x24dfba0, 0xc0008be960, 0x1, 0xc000642960)
    /go/pkg/mod/k8s.io/apimachinery@v0.19.2/pkg/util/wait/wait.go:156 +0xa3
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000925d40, 0x3b9aca00, 0x0, 0x1, 0xc000642960)
    /go/pkg/mod/k8s.io/apimachinery@v0.19.2/pkg/util/wait/wait.go:133 +0x98
k8s.io/apimachinery/pkg/util/wait.Until(0xc000925d40, 0x3b9aca00, 0xc000642960)
    /go/pkg/mod/k8s.io/apimachinery@v0.19.2/pkg/util/wait/wait.go:90 +0x4d
created by github.com/argoproj/argo-cd/controller.(*ApplicationController).Run
    /go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:474 +0x44d
@alexmt alexmt added the bug Something isn't working label Nov 30, 2020
@jessesuen jessesuen added this to the v1.9 milestone Nov 30, 2020
@alexmt
Copy link
Collaborator Author

alexmt commented Nov 30, 2020 •
edited

Most likely this is day zero issue - so ok to fix in v1.9. I think to fix it we just need to upgrade github.com/vmihailenco/msgpack/v5

alexmt added a commit to alexmt/argo-cd that referenced this issue Dec 2, 2020
@alexmt
fix: upgrades github.com/vmihailenco/msgpack/v5 to fix argoproj#4933
de6f7e8 
Signed-off-by: Alexander Matyushentsev <AMatyushentsev@gmail.com>
@alexmt
Copy link
Collaborator Author

alexmt commented Dec 2, 2020

Don't know why but we've started getting this error very frequently on one Argo CD instance. Trying to fix it in 1.8.

@alexmt alexmt modified the milestones: v1.9, v1.8 Dec 2, 2020
alexmt pushed a commit that referenced this issue Dec 2, 2020
@alexmt
fix: upgrades github.com/vmihailenco/msgpack/v5 to fix #4933 (#4952)
74a92c6 
Signed-off-by: Alexander Matyushentsev <AMatyushentsev@gmail.com>
@alexmt alexmt reopened this Dec 3, 2020
@alexmt
Copy link
Collaborator Author

alexmt commented Dec 3, 2020

still an issue after upgrading github.com/vmihailenco/msgpack/v5

@alexmt alexmt mentioned this issue Dec 3, 2020
Merged
rbreeze pushed a commit that referenced this issue Jan 12, 2021
@rbreeze
fix: upgrades github.com/vmihailenco/msgpack/v5 to fix #4933 (#4952)
630a0fb 
Signed-off-by: Alexander Matyushentsev <AMatyushentsev@gmail.com>
Signed-off-by: Remington Breeze <remington@breeze.software>
@johejo
Copy link
Contributor

johejo commented Jan 12, 2021

Is this issue reported to vmihailenco/msgpack?

@shiwano shiwano mentioned this issue Sep 1, 2021
Closed
@shiwano
Copy link

shiwano commented Sep 2, 2021

Since vmihailenco/msgpack does not use the buftool package, I think this problem is of go-redis/cache which uses the bufpool package.
I encountered the same problem, so I created an issue for this problem on the go-redis/cache repository.

@shiwano
Copy link

shiwano commented Sep 16, 2021

go-redis/cache#68 has been fixed, so probably this issue will be fixed after upgrading to github.com/go-redis/cache@v8.4.2

@alexmt alexmt reopened this Sep 16, 2021
@alexmt alexmt modified the milestones: v1.8, v2.2 Sep 16, 2021
@mayzhang2000 mayzhang2000 self-assigned this Oct 15, 2021
@mayzhang2000 mayzhang2000 mentioned this issue Oct 15, 2021
Merged
10 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mayzhang2000 mayzhang2000

Labels
bug Something isn't working
Projects
None yet
Milestone
v2.2
5 participants
@alexmt @shiwano @mayzhang2000 @jessesuen @johejo