Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tidb connection running loop panic while add and drop column #32311

Closed
Lily2025 opened this issue Feb 14, 2022 · 2 comments · Fixed by #36230
Closed

tidb connection running loop panic while add and drop column #32311

Lily2025 opened this issue Feb 14, 2022 · 2 comments · Fixed by #36230
Assignees
Labels
affects-5.4 This bug affects 5.4.x versions. affects-6.0 affects-6.1 affects-6.2 found/automation Found by automation tests severity/major sig/planner SIG: Planner type/bug This issue is a bug.

Comments

@Lily2025
Copy link

Lily2025 commented Feb 14, 2022

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

oltp_fun_002
tidb_partition_prune_mode='dynamic'
1、run workload tpcc
tableNum : 64
tableSize : 70000000
threads : 512
2、add and drop column repeatly while run workload
alter table stock add column sys_col1
alter table stock drop column sys_col1

2. What did you expect to see? (Required)

connection is normal

3. What did you see instead (Required)

connection running loop panic

6.1.0:
[2022/06/06 17:09:34.899 +00:00] [ERROR] [conn.go:1056] ["connection running loop panic"] [conn=3985160103664813973] [lastSQL="UPDATE sbtest10 SET c=? WHERE id=? [arguments: ("05967794061-60627611816-57000753803-54638960598-71932929990-92781692496-58924534734-03983813208-99775712551-92367922195", 26)]"] [err="runtime error: index out of range [4] with length 4"] [stack="github.com/pingcap/tidb/server.(*clientConn).Run.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:1059\nruntime.gopanic\n\t/usr/local/go/src/runtime/panic.go:838\ngithub.com/pingcap/tidb/executor.(*ExecStmt).Exec.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/adapter.go:371\nruntime.gopanic\n\t/usr/local/go/src/runtime/panic.go:838\nruntime.goPanicIndex\n\t/usr/local/go/src/runtime/panic.go:89\ngithub.com/pingcap/tidb/planner/core.CheckUpdateList\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/planner/core/logical_plan_builder.go:5182\ngithub.com/pingcap/tidb/executor.(*executorBuilder).buildUpdate\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/builder.go:2145\ngithub.com/pingcap/tidb/executor.(*executorBuilder).build\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/builder.go:255\ngithub.com/pingcap/tidb/executor.(*ExecuteExec).Build\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/prepared.go:304\ngithub.com/pingcap/tidb/executor.(*ExecStmt).buildExecutor\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/adapter.go:870\ngithub.com/pingcap/tidb/executor.(*ExecStmt).Exec\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/adapter.go:413\ngithub.com/pingcap/tidb/session.runStmt\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/session.go:2069\ngithub.com/pingcap/tidb/session.(*session).preparedStmtExec\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/session.go:2223\ngithub.com/pingcap/tidb/session.(*session).ExecutePreparedStmt\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/session.go:2419\ngithub.com/pingcap/tidb/server.(*TiDBStatement).Execute\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/driver_tidb.go:75\ngithub.com/pingcap/tidb/server.(*clientConn).executePreparedStmtAndWriteResult\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn_stmt.go:234\ngithub.com/pingcap/tidb/server.(*clientConn).handleStmtExecute\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn_stmt.go:201\ngithub.com/pingcap/tidb/server.(*clientConn).dispatch\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:1393\ngithub.com/pingcap/tidb/server.(*clientConn).Run\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:1121\ngithub.com/pingcap/tidb/server.(*Server).onConn\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/server.go:559"]
[2022/06/06 17:09:34.904 +00:00] [INFO] [delete_range.go:105] ["[ddl] add job into delete-range table"] [jobID=313] [jobType="drop column"]

5.4.0:
[2022/02/12 14:03:32.873 +00:00] [ERROR] [conn.go:801] ["connection running loop panic"] [conn=2673] [lastSQL="UPDATE stock SET s_quantity = ?, s_ytd = s_ytd + ?, s_order_cnt = s_order_cnt + 1, s_remote_cnt = s_remote_cnt + ? WHERE s_i_id = ? AND s_w_id = ? [arguments: (46, 2, 0, 26283, 5311)]"] [err="runtime error: index out of range [17] with length 17"] stack="goroutine 131661450 [running]:[ngithub.com/pingcap/tidb/server.(*clientConn).Run.func1(0x3c187d8, 0xc0276dfec0, 0xc00c055e00)\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/server/conn.go:799 +0xf5\npanic(0x3555700, 0xc0a2839b48)\n\t/usr/local/go/src/runtime/panic.go:965 +0x1b9[ngithub.com/pingcap/tidb/executor.(*ExecStmt).Exec.func1](http://ngithub.com/pingcap/tidb/executor.(*ExecStmt).Exec.func1)(0xc0b353cc30, 0xc0088aac40, 0xc0088aac20)\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/adapter.go:332 +0x4d4\npanic(0x3555700, 0xc0a2839b48)\n\t/usr/local/go/src/runtime/panic.go:965 +0x1b9[ngithub.com/pingcap/tidb/table/tables.(*TableCommon).UpdateRecord](http://ngithub.com/pingcap/tidb/table/tables.(*TableCommon).UpdateRecord)(0xc0bcec4000, 0x3c187d8, 0xc0322d84e0, 0x3c58130, 0xc011c5bcc0, 0x3c44ac8, 0xc09a8b9310, 0xc0b8719600, 0x11, 0x12, ...)\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/table/tables/tables.go:366 +0x2e1b[ngithub.com/pingcap/tidb/executor.updateRecord](http://ngithub.com/pingcap/tidb/executor.updateRecord)(0x3c187d8, 0xc0322d84e0, 0x3c58130, 0xc011c5bcc0, 0x3c44ac8, 0xc09a8b9310, 0xc0b8719600, 0x11, 0x12, 0xc0b8719b80, ...)\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/write.go:205 +0x1434[ngithub.com/pingcap/tidb/executor.(*UpdateExec).exec](http://ngithub.com/pingcap/tidb/executor.(*UpdateExec).exec)(0xc0a803c180, 0x3c187d8, 0xc0322d84e0, 0xc0a0ccd630, 0xc0b8719600, 0x12, 0x12, 0xc0b8719b80, 0x12, 0x12, ...)\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/update.go:188 +0x3f7[ngithub.com/pingcap/tidb/executor.(*UpdateExec).updateRows](http://ngithub.com/pingcap/tidb/executor.(*UpdateExec).updateRows)(0xc0a803c180, 0x3c187d8, 0xc0322d84e0, 0x0, 0x0, 0x0)\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/update.go:301 +0x53f[ngithub.com/pingcap/tidb/executor.(*UpdateExec).Next](http://ngithub.com/pingcap/tidb/executor.(*UpdateExec).Next)(0xc0a803c180, 0x3c187d8, 0xc0322d84e0, 0xc0a0ccd900, 0x0, 0xc0088aa730)\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/update.go:219 +0x6c[ngithub.com/pingcap/tidb/executor.Next](http://ngithub.com/pingcap/tidb/executor.Next)(0x3c187d8, 0xc0322d84e0, 0x3c1d3f8, 0xc0a803c180, 0xc0a0ccd900, 0x0, 0x0)\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/executor.go:286 +0x2de[ngithub.com/pingcap/tidb/executor.(*ExecStmt).handleNoDelayExecutor](http://ngithub.com/pingcap/tidb/executor.(*ExecStmt).handleNoDelayExecutor)(0xc0b353cc30, 0x3c187d8, 0xc0322d84e0, 0x3c1d3f8, 0xc0a803c180, 0x0, 0x0, 0x0, 0x0)\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/adapter.go:589 +0x2e7[ngithub.com/pingcap/tidb/executor.(*ExecStmt).handlePessimisticDML](http://ngithub.com/pingcap/tidb/executor.(*ExecStmt).handlePessimisticDML)(0xc0b353cc30, 0x3c187d8, 0xc0322d84e0, 0x3c1d3f8, 0xc0a803c180, 0xc07a0d6d340d7e1a, 0xb8e72917ece)\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/adapter.go:608 +0x185[ngithub.com/pingcap/tidb/executor.(*ExecStmt).handleNoDelay](http://ngithub.com/pingcap/tidb/executor.(*ExecStmt).handleNoDelay)(0xc0b353cc30, 0x3c187d8, 0xc0322d84e0, 0x3c1d3f8, 0xc0a803c180, 0x57b8e01, 0x3c18701, 0x0, 0x0, 0x0, ...)\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/adapter.go:468 +0x155[ngithub.com/pingcap/tidb/executor.(*ExecStmt).Exec](http://ngithub.com/pingcap/tidb/executor.(*ExecStmt).Exec)(0xc0b353cc30, 0x3c187d8, 0xc0322d84e0, 0x0, 0x0, 0x0, 0x0)\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/executor/adapter.go:419 +0x707[ngithub.com/pingcap/tidb/session.runStmt](http://ngithub.com/pingcap/tidb/session.runStmt)(0x3c187d8, 0xc09c759980, 0xc011c5bcc0, 0x3c2e638, 0xc0b353cc30, 0x0, 0x0, 0x0, 0x0)\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/session.go:1637 +0x37f[ngithub.com/pingcap/tidb/session.(*session).preparedStmtExec](http://ngithub.com/pingcap/tidb/session.(*session).preparedStmtExec)(0xc011c5bcc0, 0x3c187d8, 0xc09c759980, 0x3c4eea8, 0xc001b7c730, 0x0, 0x6, 0xc00d01ec80, 0xc009fee600, 0x5, ...)\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/session.go:1774 +0x299[ngithub.com/pingcap/tidb/session.(*session).ExecutePreparedStmt](http://ngithub.com/pingcap/tidb/session.(*session).ExecutePreparedStmt)(0xc011c5bcc0, 0x3c187d8, 0xc09c759980, 0x6, 0xc009fee600, 0x5, 0x5, 0x0, 0x0, 0x0, ...)\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/session/session.go:1935 +0x458[ngithub.com/pingcap/tidb/server.(*TiDBStatement).Execute](http://ngithub.com/pingcap/tidb/server.(*TiDBStatement).Execute)(0xc019fc4540, 0x3c187d8, 0xc09c759980, 0xc009fee600, 0x5, 0x5, 0x7fec76b2c0c0, 0xc000d87000, 0x0, 0xc0088aaff0)\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingc"]

4. What is your TiDB version? (Required)

[2022/02/12 18:39:26.113 +08:00] [INFO] [client.go:338] ["Cluster version information"] [type=tikv] [version=5.1.3] [git_hash=937df38124b977518eb2b932d7ced8be5d4eb163]
[2022/02/12 18:39:26.113 +08:00] [INFO] [client.go:338] ["Cluster version information"] [type=tidb] [version=5.1.4] [git_hash=094b3e5e69d0921e2abe6907d217478bb5a7082d]
[2022/02/12 18:39:26.113 +08:00] [INFO] [client.go:338] ["Cluster version information"] [type=pd] [version=5.1.3] [git_hash=6919bc70f8211f8cd9b46e1d5e9d8058980ffa16]

@Lily2025 Lily2025 added the type/bug This issue is a bug. label Feb 14, 2022
@Lily2025
Copy link
Author

/assign winoros

@Lily2025
Copy link
Author

/severity major

@ti-chi-bot ti-chi-bot added severity/major may-affects-4.0 This bug maybe affects 4.0.x versions. may-affects-5.0 This bug maybe affects 5.0.x versions. may-affects-5.1 This bug maybe affects 5.1.x versions. may-affects-5.2 This bug maybe affects 5.2.x versions. may-affects-5.3 This bug maybe affects 5.3.x versions. may-affects-5.4 This bug maybe affects 5.4.x versions. labels Feb 14, 2022
@Lily2025 Lily2025 added the found/automation Found by automation tests label Feb 14, 2022
@aytrack aytrack added the sig/execution SIG execution label Feb 14, 2022
@jebter jebter added the affects-5.1 This bug affects 5.1.x versions. label Feb 14, 2022
@ti-chi-bot ti-chi-bot removed the may-affects-5.1 This bug maybe affects 5.1.x versions. label Feb 14, 2022
@chrysan chrysan added the sig/planner SIG: Planner label Jun 7, 2022
@zanmato1984 zanmato1984 removed the sig/execution SIG execution label Jun 15, 2022
@Yisaer Yisaer added affects-5.4 This bug affects 5.4.x versions. and removed affects-5.1 This bug affects 5.1.x versions. may-affects-4.0 This bug maybe affects 4.0.x versions. may-affects-5.2 This bug maybe affects 5.2.x versions. may-affects-5.3 This bug maybe affects 5.3.x versions. may-affects-5.0 This bug maybe affects 5.0.x versions. labels Jul 15, 2022
@ti-chi-bot ti-chi-bot removed the may-affects-5.4 This bug maybe affects 5.4.x versions. label Jul 15, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
affects-5.4 This bug affects 5.4.x versions. affects-6.0 affects-6.1 affects-6.2 found/automation Found by automation tests severity/major sig/planner SIG: Planner type/bug This issue is a bug.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

9 participants