SIGSEGV crash in RequestScale

[alt-dima]

Report

Keda crashes in the middle of the operations

Expected Behavior

Not to crash and continue operations

Actual Behavior

Crash with panic: runtime error: invalid memory address or nil pointer dereference

Steps to Reproduce the Problem

Can't reproduce, it happens in the middle of the regular operations.
Will try to find and correlations.
We have 3 EKS clusters with Keda. And Keda crashes only in the biggest cluster (with 1700 pods).

Logs from KEDA operator

2023-05-21T05:31:44Z    INFO    Reconciling ScaledObject    {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"find-reservation-sources","namespace":"default"}, "namespace": "default"
, "name": "find-reservation-sources", "reconcileID": "357b528d-5fd7-488a-809e-d5ff382b3c75"}                                                                                                                                                                
2023-05-21T05:31:44Z    INFO    Reconciling ScaledObject    {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"api","namespace":"default"}, "namespace": "default", "name": "api", "rec
oncileID": "0d4960df-8e8f-4233-82ee-b3aa03491a6e"}                                                                                                                                                                                                          
2023-05-21T05:31:44Z    INFO    scaleexecutor    Successfully updated ScaleTarget    {"scaledobject.Name": "communication-attachments-worker", "scaledObject.Namespace": "default", "scaleTarget.Name": "communication-attachments-worker", "Original Replic
as Count": 0, "New Replicas Count": 1}                                                                                                                                                                                                                      
panic: runtime error: invalid memory address or nil pointer dereference                                                                                                                                                                                     
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x16fb3c4]                                                                                                                                                                                     
                                                                                                                                                                                                                                                            
goroutine 8740 [running]:                                                                                                                                                                                                                                   
github.com/kedacore/keda/v2/pkg/scaling/executor.(*scaleExecutor).RequestScale(0xc000115c70, {0x434bd88, 0xc0062aa5c0}, 0xc006c0da00, 0x0, 0x0)                                                                                                             
    /workspace/pkg/scaling/executor/scale_scaledobjects.go:39 +0x184                                                                                                                                                                                        
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers(0xc00082ee70, {0x434bd88, 0xc0062aa5c0}, {0x3a94540?, 0xc006c0da00?}, {0x4337c18, 0xc007aa0f98})                                                                                       
    /workspace/pkg/scaling/scale_handler.go:234 +0x427                                                                                                                                                                                                      
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop(0x0?, {0x434bd88, 0xc0062aa5c0}, 0xc002845180, {0x3a94540, 0xc006c0da00}, {0x4337c18, 0xc007aa0f98})                                                                                 
    /workspace/pkg/scaling/scale_handler.go:167 +0x351                                                                                                                                                                                                      
created by github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).HandleScalableObject                                                                                                                                                                     
    /workspace/pkg/scaling/scale_handler.go:122 +0x55d

KEDA Version

2.10.0

Kubernetes Version

1.24

Platform

Amazon Web Services

Scaler Details

CPU, Memory, Kafka, RabbitMQ

Anything else?

No response

[alt-dima]

Updating to version 2.10.1 to check if it will help

[tobotg]

I guess this is related to #4413 and #4389

[alt-dima]

With version 2.10.1 it works without crash for a week already.
probably I will close this issue next week

[zroubalik]

I guess this is related to #4413 and #4389

I think this was is a different issue (fixed by 2.10.1 patch)

@alt-dima thanks for letting us know! Please keep us posted.

[djsly]

@zroubalik are you certain that both #4413 and #4389 are fixed in 2.10.1 ? why isn't there a matching PR on them which shows the fix and why are they still opened ?

[zroubalik]

@zroubalik are you certain that both #4413 and #4389 are fixed in 2.10.1 ? why isn't there a matching PR on them which shows the fix and why are they still opened ?

No these ^^ are not fixed yet.

I belive the issue described here has been fixed in 2.10.1

[alt-dima]

Closing this issue. Keda pods are up for 11 days without crash