[Bug]: tpcc TT_PAYMENT txn report internal error: panic runtime error: invalid memory address or nil pointer dereference during tpcc statbility test on distributed mode

[aressu1985]

Is there an existing issue for the same bug?

• I have checked the existing issues.

Branch Name

1.2-dev

Commit ID

115aa1f

Other Environment Information

- Hardware parameters:
3*CN: 16C 64G
1*DN: 16C 64G
3*LOG: 4C 16G
3*PROXY 3C 7G
- OS type:
- Others:

Actual Behavior

during tpcc statbility test on distributed mode, there was sometimes one error in txn TT_PAYMENT:
2024-07-06 05:23:08 FATAL jTPCCTerminal:214 - [UNEXPECTED][TT_PAYMENT][EXECUTION] ErrorCode : 20101, ErrorMessage : internal error: panic runtime error: invalid memory address or nil pointer dereference:
runtime.panicmem
/usr/local/go/src/runtime/panic.go:261
runtime.sigpanic
/usr/local/go/src/runtime/signal_unix.go:881
github.com/matrixorigin/matrixone/pkg/perfcounter.WithCounterSet
/go/src/github.com/matrixorigin/matrixone/pkg/perfcounter/context.go:27
github.com/matrixorigin/matrixone/pkg/sql/compile.(*Compile).Compile
/go/src/github.com/matrixorigin/matrixone/pkg/sql/compile/compile.go:273
github.com/matrixorigin/matrixone/pkg/sql/compile.(*Compile).prepareRetry
/go/src/github.com/matrixorigin/matrixone/pkg/sql/compile/compile.go:590
github.com/matrixorigin/matrixone/pkg/sql/compile.(*Compile).Run
/go/src/github.com/matrixorigin/matrixone/pkg/sql/compile/compile.go:541
github.com/matrixorigin/matrixone/pkg/frontend.executeStatusStmt
/go/src/github.com/matrixorigin/matrixone/pkg/frontend/status_stmt.go:136
github.com/matrixorigin/matrixone/pkg/frontend.executeStmt
/go/src/github.com/matrixorigin/matrixone/pkg/frontend/mysql_cmd_executor.go:2562
github.com/matrixorigin/matrixone/pkg/frontend.dispatchStmt
/go/src/github.com/matrixorigin/matrixone/pkg/frontend/mysql_cmd_executor.go:2426
github.com/matrixorigin/matrixone/pkg/frontend.executeStmtWithIncrStmt
/go/src/github.com/matrixorigin/matrixone/pkg/frontend/mysql_cmd_executor.go:2395
github.com/matrixorigin/matrixone/pkg/frontend.executeStmtWithWorkspace
/go/src/github.com/matrixorigin/matrixone/pkg/frontend/mysql_cmd_executor.go:2352
github.com/matrixorigin/matrixone/pkg/frontend.executeStmtWithTxn
/go/src/github.com/matrixorigin/matrixone/pkg/frontend/mysql_cmd_executor.go:2243
github.com/matrixorigin/matrixone/pkg/frontend.executeStmtWithResponse
/go/src/github.com/matrixorigin/matrixone/pkg/frontend/mysql_cmd_executor.go:2208
github.com/matrixorigin/matrixone/pkg/frontend.doComQuery
/go/src/github.com/matrixorigin/matrixone/pkg/frontend/mysql_cmd_executor.go:2791
github.com/matrixorigin/matrixone/pkg/frontend.ExecRequest
/go/src/github.com/matrixorigin/matrixone/pkg/frontend/mysql_cmd_executor.go:2924
github.com/matrixorigin/matrixone/pkg/frontend.(*Routine).handleRequest
/go/src/github.com/matrixorigin/matrixone/pkg/frontend/routine.go:291
github.com/matrixorigin/matrixone/pkg/frontend.(*RoutineManager).Handler
/go/src/github.com/matrixorigin/matrixone/pkg/frontend/routine_manager.go:542
github.com/matrixorigin/matrixone/pkg/frontend.(*MOServer).handleMessage
/go/src/github.com/matrixorigin/matrixone/pkg/frontend/server.go:205
github.com/fagongzi/goetty/v2.(*server).doStart.func2.1

mo-log:
https://shanghai.idc.matrixorigin.cn:30001/explore?panes=%7B%22CvG%22:%7B%22datasource%22:%22loki%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bnamespace%3D%5C%22mo-115aa1f-202407060241%5C%22%7D%20%7C%3D%20%60panic%60%22,%22queryType%22:%22range%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22loki%22%7D,%22editorMode%22:%22builder%22%7D%5D,%22range%22:%7B%22from%22:%221720214272978%22,%22to%22:%221720214836456%22%7D%7D%7D&schemaVersion=1&orgId=1

Expected Behavior

No response

Steps to Reproduce

1. run a mo cluster with config in this issue
2. run tpch 10G loop test processes in one independant tenant
3. run tpcc 10 warehouse and 10 ternimals longrunnig test processes in one independant tenant, prepare mode
4. run sysbench mixed cases(insert/delete/update/select) longrunnig test processes with 75 terminals in one independant tenant,non-prepare mode
5. run another sysbench mixed cases(insert/delete/update/select) longrunnig test processe with  75 terminals in one independant tenant,non-prepare mode

Additional information

TT_PAYMENT TXN:

1、PAYMENT
//每个事物有15%的概率，支付的非该事务关联的warehouse
a.
//更新distinct所有订单总金额
UPDATE bmsql_district " +
" SET d_ytd = d_ytd + ? " +
" WHERE d_w_id = ? AND d_id = ?

b.
SELECT d_name, d_street_1, d_street_2, d_city, " +
" d_state, d_zip " +
" FROM bmsql_district " +
" WHERE d_w_id = ? AND d_id = ?"
c.
//更新warehouse所有订单总金额
UPDATE bmsql_warehouse " +
" SET w_ytd = w_ytd + ? " +
" WHERE w_id = ?

d.
SELECT w_name, w_street_1, w_street_2, w_city, " +
" w_state, w_zip " +
" FROM bmsql_warehouse " +
" WHERE w_id = ? ")
e.
//60%的概率会执行该语句
SELECT c_id " +
" FROM bmsql_customer " +
" WHERE c_w_id = ? AND c_d_id = ? AND c_last = ? " +
" ORDER BY c_first")

f.
SELECT c_first, c_middle, c_last, c_street_1, c_street_2, " +
" c_city, c_state, c_zip, c_phone, c_since, c_credit, " +
" c_credit_lim, c_discount, c_balance " +
" FROM bmsql_customer " +
" WHERE c_w_id = ? AND c_d_id = ? AND c_id = ? " +
" FOR UPDATE"
h.
//如果customer with good credit
UPDATE bmsql_customer " +
" SET c_balance = c_balance - ?, " +
" c_ytd_payment = c_ytd_payment + ?, " +
" c_payment_cnt = c_payment_cnt + 1 " +
" WHERE c_w_id = ? AND c_d_id = ? AND c_id = ?"
//如果customer whith bad credit
UPDATE bmsql_customer " +
" SET c_balance = c_balance - ?, " +
" c_ytd_payment = c_ytd_payment + ?, " +
" c_payment_cnt = c_payment_cnt + 1, " +
" c_data = ? " +
" WHERE c_w_id = ? AND c_d_id = ? AND c_id = ?
i.
//更新历史记录表
INSERT INTO bmsql_history (" +
" h_c_id, h_c_d_id, h_c_w_id, h_d_id, h_w_id, " +
" h_date, h_amount, h_data) " +
"VALUES (?, ?, ?, ?, ?, ?, ?, ?)

[m-schen]

相关pr已revert，可关闭这个issue。