[Bug]: there are lots queries running more than serval hours during stabllity test on distributed mode

[aressu1985]

Is there an existing issue for the same bug?

• I have checked the existing issues.

Branch Name

1.1-dev

Commit ID

e0d9dad

Other Environment Information

- Hardware parameters:
3*CN: 16C 64G
1*DN: 16C 64G
3*LOG: 4C 16G
- OS type:
- Others:

Actual Behavior

During stability test on distributed mode, there were lots of queries running for serval hours, such as :

mo-log:
https://shanghai.idc.matrixorigin.cn:30001/explore?panes=%7B%22Ddj%22:%7B%22datasource%22:%22loki%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bnamespace%3D%5C%22mo-nightly-e0d9dad-20240411%5C%22%7D%20%7C%3D%20%60cn-service%20found%20long%20running%20txn%60%22,%22queryType%22:%22range%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22loki%22%7D,%22editorMode%22:%22builder%22%7D%5D,%22range%22:%7B%22from%22:%221712886664617%22,%22to%22:%221712909937000%22%7D%7D%7D&schemaVersion=1&orgId=1

Expected Behavior

No response

Steps to Reproduce

1. run a mo cluster using config in this issue
2. run tpch 10g loop queries in a account
3. run tpcc 10-10 longrunnig test in a account
4. run two sysbench mixed cases test in two different accounts

Additional information

No response

[zhangxu19830126]

fixed with #15514

[aressu1985]

this issue still exists on stability test on distributed mode:

mo-log:
https://shanghai-idc.1matrix.org:30088/explore?panes=%7B%224QD%22:%7B%22datasource%22:%22loki%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bnamespace%3D%5C%22mo-nightly-5c0ae1c-20240418%5C%22%7D%20%7C%3D%20%60long%20running%60%22,%22queryType%22:%22range%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22loki%22%7D,%22editorMode%22:%22builder%22%7D%5D,%22range%22:%7B%22from%22:%221713499900058%22,%22to%22:%221713542997770%22%7D%7D%7D&schemaVersion=1&orgId=1

goroutine:
Uploading routine_018ef48d-0062-7dbe-9d83-28c11619502e.txt…

[zhangxu19830126]

fixed

[aressu1985]

目前依然存在这个问题

[aressu1985]

mo-log:
https://shanghai-idc.1matrix.org:30088/explore?panes=%7B%22C6x%22:%7B%22datasource%22:%22loki%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bnamespace%3D%5C%22mo-nightly-82b931e-20240420%5C%22%7D%20%7C%3D%20%60long%20running%60%22,%22queryType%22:%22range%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22loki%22%7D,%22editorMode%22:%22builder%22%7D%5D,%22range%22:%7B%22from%22:%22now-12h%22,%22to%22:%22now%22%7D%7D%7D&schemaVersion=1&orgId=1

[aressu1985]

fixed