Tidb crashes when sql with shuffleExec exceed tidb_mem_quota_query

[wshwsh12]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

use test;
drop table if exists t1;
create table t1(i int, j int, k int);
insert into t1 VALUES (1,1,1),(2,2,2),(3,3,3),(4,4,4);
INSERT INTO t1 SELECT 10*i,j,5*j FROM t1 UNION SELECT 20*i,j,5*j FROM t1 UNION SELECT 30*i,j,5*j FROM t1;
INSERT INTO t1 SELECT 10*i,j,5*j FROM t1 UNION SELECT 20*i,j,5*j FROM t1 UNION SELECT 30*i,j,5*j FROM t1;
INSERT INTO t1 SELECT 10*i,j,5*j FROM t1 UNION SELECT 20*i,j,5*j FROM t1 UNION SELECT 30*i,j,5*j FROM t1;
set @@session.tidb_enable_window_function=1;
set tidb_window_concurrency = 128;
set tidb_mem_quota_query = 13000;

run the sql many times(more than 100000 times):
SELECT SUM(i) OVER W FROM t1 WINDOW w AS (PARTITION BY j ORDER BY i) ORDER BY 1+SUM(i) OVER w;

2. What did you expect to see? (Required)

3. What did you see instead (Required)

TiDB crashes.

panic: send on closed channel [recovered]
	panic: send on closed channel

goroutine 108646 [running]:
github.com/pingcap/tidb/pkg/executor.recoveryShuffleExec(0x62f14f0?, {0x52bf360, 0x62acbf0})
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/executor/shuffle.go:259 +0x199
github.com/pingcap/tidb/pkg/executor.(*ShuffleExec).fetchDataAndSplit.func1()
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/executor/shuffle.go:273 +0x4d
panic({0x52bf360?, 0x62acbf0?})
	/usr/local/go/src/runtime/panic.go:914 +0x21f
github.com/pingcap/tidb/pkg/executor.(*ShuffleExec).fetchDataAndSplit(0xc005cf4d20, {0x62f14f0, 0xc00763fb00}, 0x0)
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/executor/shuffle.go:283 +0x44c
created by github.com/pingcap/tidb/pkg/executor.(*ShuffleExec).prepare4ParallelExec in goroutine 20378
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/executor/shuffle.go:208 +0x48

4. What is your TiDB version? (Required)

nightly-20231101