runtime: crash with traceback stuck

[XuHuaiyu]

Go version

go1.19.12

Output of go env in your module/workspace:

$ go env

What did you do?

There has been one crash occurrence currently, with no clear reproducible method identified. The node has consistently been running daily business workloads, with no other changes in operations.

What did you see happen?

In our customer's production environment cluster, a database node crash occurred, and the error stack is as follows:

runtime: traceback stuck. pc=0x3973c50 sp=0x40024d7fb0
stack: frame={sp:0x40024d7fb0, fp:0x40024d7fb0} stack=[0x40024d4000,0x40024d8000)
0x00000040024d7eb0:  0x0001000000010000  0x000000410220f500 
0x00000040024d7ec0:  0x000000417b54c000  0x0000000004c76ee0 
0x00000040024d7ed0:  0x000000410220f500  0x000000417b4e5f08 
0x00000040024d7ee0:  0x0000000001d1d0f0 <google.golang.org/grpc/internal/transport.(*loopyWriter).run+0x0000000000000060>  0x000000417b338360 
0x00000040024d7ef0:  0x0000000000000000  0x000000417b338480 
0x00000040024d7f00:  0x0000000000000000  0x0000000000000000 
0x00000040024d7f10:  0x0000000001d321a8 <google.golang.org/grpc/internal/transport.NewServerTransport.func2+0x00000000000000d8>  0x000000410220f4f0 
0x00000040024d7f20:  0x0000000003f7cd01  0x00000040cfcf3c90 
0x00000040024d7f30:  0x0000000003973c1c <github.com/pingcap/tidb/executor.(*IndexLookUpExecutor).startTableWorker.func1+0x00000000000000bc>  0x01000040024ecf68 
0x00000040024d7f40:  0x0000000000000000  0x0000000000000000 
0x00000040024d7f50:  0x0000000001d1d330 <google.golang.org/grpc/internal/transport.(*loopyWriter).run.func1+0x0000000000000000>  0x00000040024d7f40 
0x00000040024d7f60:  0x00000040024d7f50  0x0000000000000000 
0x00000040024d7f70:  0x000000000187de34 <runtime.goexit+0x0000000000000004>  0x000000417b391b60 
0x00000040024d7f80:  0x000000417b319850  0x000000410220f4f0 
0x00000040024d7f90:  0x0000000000000000  0x0000004173a86500 
0x00000040024d7fa0:  0x000000417ac4d000  0x000000417a51d040 
0x00000040024d7fb0: >0x0000000003973c50 <github.com/pingcap/tidb/executor.(*IndexLookUpExecutor).startTableWorker.func1.1+0x0000000000000000>  0x000000417b348000 
0x00000040024d7fc0:  0x00000040024ecfb0  0x0000000000000000 
0x00000040024d7fd0:  0x0000000000000000  0x0000000000000000 
0x00000040024d7fe0:  0x0000000000000000  0x0000000000000000 
0x00000040024d7ff0:  0x0000000000000000  0x0000000000000000 
fatal error: traceback stuck

What did you expect to see?

No crash

[dr2chase]

Is this a one-time failure, or does it happen more often?

Upgrading to the current version of Go (1.22) might help, and also has a better chance of a fix because we are no longer fixing bugs in 1.19.

[XuHuaiyu]

@dr2chase
Thank you for your reply. This is a one-time bug. However, we are unable to recompile and upgrade the services in the user's environment. Are there any known bugs in the version we are using that could potentially cause this issue? I tried searching for the error message keyword but couldn't find any relevant issues.

[gopherbot]

Timed out in state WaitingForInfo. Closing.

(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)