Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Avoid accessing readyPodQueue if not initialized #2258

Closed
sanketsudake opened this issue Nov 9, 2021 · 0 comments
Closed

Avoid accessing readyPodQueue if not initialized #2258

sanketsudake opened this issue Nov 9, 2021 · 0 comments
Labels
area-executor executor-poolmgr
Milestone
1.15.0

Comments

@sanketsudake
Copy link
Member

sanketsudake commented Nov 9, 2021 •
edited

Observed in run https://github.com/fission/fission/actions/runs/1439179980

.dumps/1439179980/kind-logs-1439179980-v1.20.7/kind-control-plane

fission/pkg/executor/executortype/poolmgr/gp.go

Line 242 in 2c1b459

item, quit := gp.readyPodQueue.Get() 

2021-11-09T11:34:57.319515074Z stderr F panic: runtime error: invalid memory address or nil pointer dereference
2021-11-09T11:34:57.319538974Z stderr F [signal SIGSEGV: segmentation violation code=0x1 addr=0x30 pc=0x1f9bbe3]
2021-11-09T11:34:57.319543375Z stderr F 
2021-11-09T11:34:57.319547675Z stderr F goroutine 357 [running]:
2021-11-09T11:34:57.319552175Z stderr F github.com/fission/fission/pkg/executor/executortype/poolmgr.(*GenericPool).choosePod(0xc00062c2d0, 0x2a11b00, 0xc0002d7aa0, 0xc000573bc0, 0x0, 0x0, 0x212e380, 0xc0006e5500, 0xc0004f6560)
2021-11-09T11:34:57.319556275Z stderr F 	pkg/executor/executortype/poolmgr/gp.go:242 +0x163
2021-11-09T11:34:57.319560575Z stderr F github.com/fission/fission/pkg/executor/executortype/poolmgr.(*GenericPool).getFuncSvc(0xc00062c2d0, 0x2a11b00, 0xc0002d7aa0, 0xc00132e6c0, 0x1, 0x1, 0x0)
2021-11-09T11:34:57.319574476Z stderr F 	pkg/executor/executortype/poolmgr/gp.go:480 +0x505
2021-11-09T11:34:57.319578676Z stderr F github.com/fission/fission/pkg/executor/executortype/poolmgr.(*GenericPoolManager).GetFuncSvc(0xc0004e6c80, 0x2a11b00, 0xc0002d7aa0, 0xc00132e6c0, 0xc00035bee8, 0x1, 0x4)
2021-11-09T11:34:57.319581976Z stderr F 	pkg/executor/executortype/poolmgr/gpm.go:197 +0x41a
2021-11-09T11:34:57.319585276Z stderr F github.com/fission/fission/pkg/executor.(*Executor).createServiceForFunction(0xc0002a85a0, 0x2a11b00, 0xc0002d7aa0, 0xc00132e6c0, 0xc0002d7aa0, 0xc0006e55d0, 0x0)
2021-11-09T11:34:57.319588976Z stderr F 	pkg/executor/executor.go:230 +0x378
2021-11-09T11:34:57.319592477Z stderr F github.com/fission/fission/pkg/executor.(*Executor).serveCreateFuncServices.func1(0xc000e18080, 0xc0002a85a0)
2021-11-09T11:34:57.319595677Z stderr F 	pkg/executor/executor.go:134 +0xb6
2021-11-09T11:34:57.319599077Z stderr F created by github.com/fission/fission/pkg/executor.(*Executor).serveCreateFuncServices
2021-11-09T11:34:57.319602677Z stderr F 	pkg/executor/executor.go:119 +0x387
sanketsudake added a commit that referenced this issue Nov 10, 2021
@sanketsudake
Issue #2258 Reafctor ready pod conroller to use lister and cache sync…
6279caa 
… checks

Reactored ready pod controller code to user lister and cache sync checks,
so that we avoid querying lister if cache is not synced in choodPod function.
Also, as noticied in #2258 we were initializing workqueue in goroutine
which was causing nil pointer reference. We have moved it out of goroutine
and kept specific parts in goroutine.

Signed-off-by: Sanket Sudake <sanketsudake@gmail.com>
@sanketsudake sanketsudake mentioned this issue Nov 10, 2021
Merged
sanketsudake added a commit that referenced this issue Nov 10, 2021
@sanketsudake
Issue #2258 Reafctor ready pod conroller to use lister and cache sync…
81e247e 
… checks (#2259)

Reactored ready pod controller code to user lister and cache sync checks,
so that we avoid querying lister if cache is not synced in choodPod function.
Also, as noticied in #2258 we were initializing workqueue in goroutine
which was causing nil pointer reference. We have moved it out of goroutine
and kept specific parts in goroutine.

Signed-off-by: Sanket Sudake <sanketsudake@gmail.com>
@sanketsudake sanketsudake added this to the 1.15.0 milestone Nov 10, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area-executor executor-poolmgr
Projects
None yet
Milestone
1.15.0
Development

No branches or pull requests
1 participant
@sanketsudake