Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: ut TestLockSuccWithKeepBindTimeout failed #15757

Open
1 task done
sukki37 opened this issue Apr 26, 2024 · 13 comments
Open
1 task done

[Bug]: ut TestLockSuccWithKeepBindTimeout failed #15757

sukki37 opened this issue Apr 26, 2024 · 13 comments
Assignees
@aressu1985
Labels
bug/ut kind/bug Something isn't working phase/testing severity/s0 Extreme impact: Cause the application to break down and seriously affect the use
Milestone
1.3.0

Comments

@sukki37
Copy link
Contributor

sukki37 commented Apr 26, 2024

Is there an existing issue for the same bug?

  • I have checked the existing issues.

Branch Name

main

Commit ID

pr

Other Environment Information

- Hardware parameters:
- OS type:
- Others:

Actual Behavior

https://github.com/matrixorigin/matrixone/actions/runs/8850698515/job/24305581063?pr=15691

2024-04-26T16:04:10.4020160Z panic: missing free for type: *lockservice.waiter, 03-0xc0005c4600(1) 
2024-04-26T16:04:10.4020281Z  create by: <<<>>>
2024-04-26T16:04:10.4020294Z 
2024-04-26T16:04:10.4020300Z 
2024-04-26T16:04:10.4020423Z goroutine 5 [running]:
2024-04-26T16:04:10.4020832Z github.com/matrixorigin/matrixone/pkg/common/reuse.(*checker[...]).gc(0x11f1760, 0xc0005c4600)
2024-04-26T16:04:10.4021219Z 	/home/runner/work/matrixone/matrixone/pkg/common/reuse/checker.go:133 +0x394
2024-04-26T16:04:10.4021572Z github.com/matrixorigin/matrixone/pkg/common/reuse.newSyncPoolBased[...].func1.1()
2024-04-26T16:04:10.4021958Z 	/home/runner/work/matrixone/matrixone/pkg/common/reuse/sync_pool_based.go:46 +0xc5
2024-04-26T16:04:10.4022214Z FAIL	github.com/matrixorigin/matrixone/pkg/lockservice	187.292s

Expected Behavior

No response

Steps to Reproduce

ut

Additional information

No response
@sukki37 sukki37 added kind/bug Something isn't working severity/s0 Extreme impact: Cause the application to break down and seriously affect the use bug/ut labels Apr 26, 2024
@matrix-meow matrix-meow added this to the 1.2.0 milestone Apr 26, 2024
@iamlinjunhong
Copy link
Contributor

处理中

@sukki37
Copy link
Contributor Author

sukki37 commented May 7, 2024

repro：https://github.com/matrixorigin/matrixone/actions/runs/8986398280/job/24682403725?pr=15880

@iamlinjunhong iamlinjunhong mentioned this issue May 8, 2024
Closed
7 tasks
@aressu1985 aressu1985 modified the milestones: 1.2.0, 1.2.1 May 8, 2024
@daviszhen
Copy link
Contributor

https://github.com/matrixorigin/matrixone/actions/runs/9050329457/job/24865567531?pr=16040

Screenshot from 2024-05-12 19-41-25

@gouhongshen
Copy link
Contributor

gouhongshen commented May 13, 2024 •
edited
Loading

repor: https://github.com/matrixorigin/matrixone/actions/runs/9057918105/job/24882671908?pr=16046

2024-05-13T06:24:01.2595852Z === RUN   TestLockSuccWithKeepBindTimeout
2024-05-13T06:24:01.2597219Z 2024/05/13 06:02:41.239813 +0000 DEBUG cn-service.lockservice.lock-server application accept loop started {"uuid": "", "name": "lock-server", "listen-addresses": "[/tmp/service-237485116-s1.sock]"}
2024-05-13T06:24:01.2599013Z 2024/05/13 06:02:41.240189 +0000 DEBUG cn-service.lockservice.lock-server application accept loop stopped {"uuid": "", "name": "lock-server", "listen-addresses": "[/tmp/service-237485116-s1.sock]"}
2024-05-13T06:24:01.2600722Z 2024/05/13 06:02:41.240303 +0000 DEBUG cn-service.lockservice.lock-server application started {"uuid": "", "name": "lock-server", "listen-addresses": "[/tmp/service-237485116-s1.sock]"}
2024-05-13T06:24:01.2601969Z 2024/05/13 06:02:41.239927 +0000 INFO cn-service.lockservice dead lock checker {"uuid": ""}
2024-05-13T06:24:01.2603053Z 2024/05/13 06:02:41.239931 +0000 INFO cn-service.lockservice.lock-client.lock-client gc idle backends task started {"uuid": ""}
2024-05-13T06:24:01.2604143Z 2024/05/13 06:02:41.239977 +0000 INFO cn-service.lockservice keep lock table bind task {"uuid": ""}
2024-05-13T06:24:01.2605038Z 2024/05/13 06:02:41.240013 +0000 INFO cn-service.lockservice dead lock checker {"uuid": ""}
2024-05-13T06:24:01.2605935Z 2024/05/13 06:02:41.240045 +0000 INFO cn-service.lockservice keep remote locks task {"uuid": ""}
2024-05-13T06:24:01.2606819Z 2024/05/13 06:02:41.240065 +0000 INFO cn-service.lockservice dead lock checker {"uuid": ""}
2024-05-13T06:24:01.2607671Z 2024/05/13 06:02:41.240101 +0000 INFO cn-service.lockservice dead lock checker {"uuid": ""}
2024-05-13T06:24:01.2608767Z 2024/05/13 06:02:41.240094 +0000 DEBUG cn-service.lockservice.lock-client.lock-client gc inactive backends task started {"uuid": ""}
2024-05-13T06:24:01.2610077Z 2024/05/13 06:02:41.240771 +0000 INFO cn-service.lockservice lock service start successfully {"uuid": "", "serviceID": "1715580161237546550s1"}
2024-05-13T06:24:01.2611381Z 2024/05/13 06:02:41.241038 +0000 DEBUG cn-service.lockservice.lock-client.lock-client gc inactive backends task started {"uuid": ""}
2024-05-13T06:24:01.2612793Z 2024/05/13 06:02:41.241046 +0000 INFO cn-service.lockservice.lock-client.lock-client gc idle backends task started {"uuid": ""}
2024-05-13T06:24:01.2613950Z 2024/05/13 06:02:41.241212 +0000 INFO cn-service.lockservice.allocator clean cannot commit task {"uuid": ""}
2024-05-13T06:24:01.2615345Z 2024/05/13 06:02:41.246064 +0000 DEBUG cn-service.lockservice.lock-server application accept loop started {"uuid": "", "name": "lock-server", "listen-addresses": "[/tmp/237385731.sock]"}
2024-05-13T06:24:01.2617017Z 2024/05/13 06:02:41.246158 +0000 DEBUG cn-service.lockservice.lock-server application accept loop stopped {"uuid": "", "name": "lock-server", "listen-addresses": "[/tmp/237385731.sock]"}
2024-05-13T06:24:01.2618626Z 2024/05/13 06:02:41.246203 +0000 DEBUG cn-service.lockservice.lock-server application started {"uuid": "", "name": "lock-server", "listen-addresses": "[/tmp/237385731.sock]"}
2024-05-13T06:24:01.2619890Z 2024/05/13 06:02:41.246296 +0000 INFO cn-service.lockservice lock allocator start successfully {"uuid": ""}
2024-05-13T06:24:01.2620903Z 2024/05/13 06:02:41.246379 +0000 DEBUG cn-service.lockservice txn created {"uuid": "", "txn": "74786e31()"}
2024-05-13T06:24:01.2622350Z 2024/05/13 06:02:41.247118 +0000 DEBUG cn-service.lockservice.lock-client start connect to remote {"uuid": "", "remote": "unix:///tmp/237385731.sock", "backend-id": "018f708b-ccde-77e6-98bb-a9a6202332f0"}
2024-05-13T06:24:01.2623927Z 2024/05/13 06:02:41.247372 +0000 DEBUG cn-service.lockservice.lock-server session init completed {"uuid": "", "name": "lock-server", "session-id": 1}
2024-05-13T06:24:01.2625622Z 2024/05/13 06:02:41.247483 +0000 DEBUG cn-service.lockservice.lock-server session connected {"uuid": "", "name": "lock-server", "listen-addresses": "[/tmp/237385731.sock]", "session-id": 1, "addr": "@"}
2024-05-13T06:24:01.2627489Z 2024/05/13 06:02:41.247606 +0000 DEBUG cn-service.lockservice.lock-client session init completed {"uuid": "", "remote": "unix:///tmp/237385731.sock", "backend-id": "018f708b-ccde-77e6-98bb-a9a6202332f0", "session-id": 0}
2024-05-13T06:24:01.2629444Z 2024/05/13 06:02:41.247688 +0000 DEBUG cn-service.lockservice.lock-client connect to remote succeed {"uuid": "", "remote": "unix:///tmp/237385731.sock", "backend-id": "018f708b-ccde-77e6-98bb-a9a6202332f0"}
2024-05-13T06:24:01.2631221Z 2024/05/13 06:02:41.247781 +0000 DEBUG cn-service.lockservice.lock-client read loop started {"uuid": "", "remote": "unix:///tmp/237385731.sock", "backend-id": "018f708b-ccde-77e6-98bb-a9a6202332f0"}
2024-05-13T06:24:01.2632960Z 2024/05/13 06:02:41.247919 +0000 DEBUG cn-service.lockservice.lock-client write loop started {"uuid": "", "remote": "unix:///tmp/237385731.sock", "backend-id": "018f708b-ccde-77e6-98bb-a9a6202332f0"}
2024-05-13T06:24:01.2634988Z 2024/05/13 06:02:41.248054 +0000 DEBUG cn-service.lockservice.lock-client write request {"uuid": "", "remote": "unix:///tmp/237385731.sock", "backend-id": "018f708b-ccde-77e6-98bb-a9a6202332f0", "request-id": 1, "request": "1: GetBind/1715580161237546550s1-0"}
2024-05-13T06:24:01.2637180Z 2024/05/13 06:02:41.248219 +0000 DEBUG cn-service.lockservice.lock-server session read message {"uuid": "", "name": "lock-server", "listen-addresses": "[/tmp/237385731.sock]", "session-id": 1, "addr": "@", "seqence": 1}
2024-05-13T06:24:01.2639113Z 2024/05/13 06:02:41.248295 +0000 DEBUG cn-service.lockservice.lock-server received request {"uuid": "", "name": "lock-server", "sequence": 1, "client": "@", "request-id": 1, "request": "1: GetBind/1715580161237546550s1-0"}
2024-05-13T06:24:01.2640644Z 2024/05/13 06:02:41.248398 +0000 DEBUG cn-service.lockservice received a request {"uuid": "", "request": "1: GetBind/1715580161237546550s1-0"}
2024-05-13T06:24:01.2642125Z 2024/05/13 06:02:41.248444 +0000 DEBUG cn-service.lockservice.lock-server handle request completed {"uuid": "", "name": "lock-server", "sequence": 1, "client": "@", "request-id": 1}
2024-05-13T06:24:01.2643674Z 2024/05/13 06:02:41.249297 +0000 INFO cn-service.lockservice.allocator bind created {"uuid": "", "table": 0, "service": "1715580161237546550s1"}
2024-05-13T06:24:01.2645080Z 2024/05/13 06:02:41.249362 +0000 DEBUG cn-service.lockservice handle request completed {"uuid": "", "response": "1: GetBind/0-1715580161237546550s1-1"}
2024-05-13T06:24:01.2646661Z 2024/05/13 06:02:41.249440 +0000 DEBUG cn-service.lockservice.lock-server write responses {"uuid": "", "name": "lock-server", "client": "@", "request-id": 1, "response": "1: GetBind/0-1715580161237546550s1-1"}
2024-05-13T06:24:01.2648708Z 2024/05/13 06:02:41.249595 +0000 DEBUG cn-service.lockservice.lock-client read response {"uuid": "", "remote": "unix:///tmp/237385731.sock", "backend-id": "018f708b-ccde-77e6-98bb-a9a6202332f0", "request-id": 1, "response": "1: GetBind/0-1715580161237546550s1-1"}
2024-05-13T06:24:01.2650561Z 2024/05/13 06:02:41.249749 +0000 INFO cn-service.lockservice bind created {"uuid": "", "service": "1715580161237546550s1", "remote": false, "bind": "0-1715580161237546550s1-1"}
2024-05-13T06:24:01.2652051Z 2024/05/13 06:02:41.249846 +0000 DEBUG cn-service.lockservice try to lock on local {"uuid": "", "txn": "74786e31()", "table": 0, "rows": "[01]", "opts": "Exclusive-Row-Wait"}
2024-05-13T06:24:01.2653744Z 2024/05/13 06:02:41.249947 +0000 DEBUG cn-service.lockservice holder added {"uuid": "", "txn": "74786e31()", "table": 0, "rows": "[01]", "opts": "Exclusive-Row-Wait", "holders": "[74786e31(1715580161237546550s1)]", "waiters": "[]"}
2024-05-13T06:24:01.2655213Z 2024/05/13 06:02:41.250040 +0000 DEBUG cn-service.lockservice lock added to txn {"uuid": "", "txn": "74786e31()", "rows": "[01]"}
2024-05-13T06:24:01.2656635Z 2024/05/13 06:02:41.250119 +0000 DEBUG cn-service.lockservice lock added to local {"uuid": "", "txn": "74786e31()", "table": 0, "rows": "[01]", "opts": "Exclusive-Row-Wait"}
2024-05-13T06:24:01.2657922Z 2024/05/13 06:02:41.250234 +0000 INFO cn-service.lockservice keep lock table bind task {"uuid": "", "cost": "9.735208ms"}
2024-05-13T06:24:01.2659123Z 2024/05/13 06:02:41.250317 +0000 INFO cn-service.lockservice keep remote locks task {"uuid": "", "cost": "9.791783ms"}
2024-05-13T06:24:01.2660027Z panic: missing free for type: *lockservice.waiter, 02-0xc000000900(1) 
2024-05-13T06:24:01.2660547Z  create by: <<<>>>
2024-05-13T06:24:01.2660721Z 
2024-05-13T06:24:01.2660726Z 
2024-05-13T06:24:01.2660862Z goroutine 5 [running]:
2024-05-13T06:24:01.2661470Z github.com/matrixorigin/matrixone/pkg/common/reuse.(*checker[...]).gc(0x1215c40, 0xc000000900)
2024-05-13T06:24:01.2662357Z 	/home/runner/work/matrixone/matrixone/pkg/common/reuse/checker.go:133 +0x394
2024-05-13T06:24:01.2663177Z github.com/matrixorigin/matrixone/pkg/common/reuse.newSyncPoolBased[...].func1.1()
2024-05-13T06:24:01.2664054Z 	/home/runner/work/matrixone/matrixone/pkg/common/reuse/sync_pool_based.go:46 +0xc5
2024-05-13T06:24:01.2664831Z FAIL	github.com/matrixorigin/matrixone/pkg/lockservice	184.230s

@xzxiong
Copy link
Contributor

xzxiong commented May 13, 2024

new case: TestReLockSuccWithLockTableBindChanged, same error.

https://github.com/matrixorigin/matrixone/actions/runs/9059936309/job/24888593773?pr=16067

2024-05-13T09:15:25.7200138Z === RUN   TestReLockSuccWithLockTableBindChanged
2024-05-13T09:15:25.7200622Z 2024/05/13 08:54:18.119309 +0000 INFO cn-service.lockservice dead lock checker {"uuid": ""}
2024-05-13T09:15:25.7201020Z 2024/05/13 08:54:18.119440 +0000 INFO cn-service.lockservice dead lock checker {"uuid": ""}
2024-05-13T09:15:25.7201453Z 2024/05/13 08:54:18.119600 +0000 INFO cn-service.lockservice dead lock checker {"uuid": ""}
2024-05-13T09:15:25.7201851Z 2024/05/13 08:54:18.119674 +0000 INFO cn-service.lockservice dead lock checker {"uuid": ""}
2024-05-13T09:15:25.7202465Z 2024/05/13 08:54:18.119811 +0000 INFO cn-service.lockservice.lock-client.lock-client gc idle backends task started {"uuid": ""}
2024-05-13T09:15:25.7203098Z 2024/05/13 08:54:18.119968 +0000 DEBUG cn-service.lockservice.lock-client.lock-client gc inactive backends task started {"uuid": ""}
2024-05-13T09:15:25.7203545Z 2024/05/13 08:54:18.120739 +0000 INFO cn-service.lockservice keep lock table bind task {"uuid": ""}
2024-05-13T09:15:25.7204036Z 2024/05/13 08:54:18.120927 +0000 INFO cn-service.lockservice keep remote locks task {"uuid": ""}
2024-05-13T09:15:25.7204945Z 2024/05/13 08:54:18.121109 +0000 DEBUG cn-service.lockservice.lock-server application accept loop started {"uuid": "", "name": "lock-server", "listen-addresses": "[/tmp/service-105352713-s1.sock]"}
2024-05-13T09:15:25.7205905Z 2024/05/13 08:54:18.126031 +0000 DEBUG cn-service.lockservice.lock-server application accept loop stopped {"uuid": "", "name": "lock-server", "listen-addresses": "[/tmp/service-105352713-s1.sock]"}
2024-05-13T09:15:25.7206740Z 2024/05/13 08:54:18.126084 +0000 DEBUG cn-service.lockservice.lock-server application started {"uuid": "", "name": "lock-server", "listen-addresses": "[/tmp/service-105352713-s1.sock]"}
2024-05-13T09:15:25.7207396Z 2024/05/13 08:54:18.126409 +0000 INFO cn-service.lockservice lock service start successfully {"uuid": "", "serviceID": "1715590458105442711s1"}
2024-05-13T09:15:25.7207791Z 2024/05/13 08:54:18.127983 +0000 INFO cn-service.lockservice dead lock checker {"uuid": ""}
2024-05-13T09:15:25.7208195Z 2024/05/13 08:54:18.134072 +0000 INFO cn-service.lockservice dead lock checker {"uuid": ""}
2024-05-13T09:15:25.7208588Z 2024/05/13 08:54:18.134173 +0000 INFO cn-service.lockservice dead lock checker {"uuid": ""}
2024-05-13T09:15:25.7208997Z 2024/05/13 08:54:18.134329 +0000 INFO cn-service.lockservice dead lock checker {"uuid": ""}
2024-05-13T09:15:25.7209603Z 2024/05/13 08:54:18.134520 +0000 INFO cn-service.lockservice.lock-client.lock-client gc idle backends task started {"uuid": ""}
2024-05-13T09:15:25.7210247Z 2024/05/13 08:54:18.134683 +0000 DEBUG cn-service.lockservice.lock-client.lock-client gc inactive backends task started {"uuid": ""}
2024-05-13T09:15:25.7210677Z 2024/05/13 08:54:18.135473 +0000 INFO cn-service.lockservice keep remote locks task {"uuid": ""}
2024-05-13T09:15:25.7211116Z 2024/05/13 08:54:18.135687 +0000 INFO cn-service.lockservice keep lock table bind task {"uuid": ""}
2024-05-13T09:15:25.7212080Z 2024/05/13 08:54:18.135894 +0000 DEBUG cn-service.lockservice.lock-server application accept loop started {"uuid": "", "name": "lock-server", "listen-addresses": "[/tmp/service-105390654-s2.sock]"}
2024-05-13T09:15:25.7212979Z 2024/05/13 08:54:18.135976 +0000 DEBUG cn-service.lockservice.lock-server application accept loop stopped {"uuid": "", "name": "lock-server", "listen-addresses": "[/tmp/service-105390654-s2.sock]"}
2024-05-13T09:15:25.7213815Z 2024/05/13 08:54:18.136044 +0000 DEBUG cn-service.lockservice.lock-server application started {"uuid": "", "name": "lock-server", "listen-addresses": "[/tmp/service-105390654-s2.sock]"}
2024-05-13T09:15:25.7214465Z 2024/05/13 08:54:18.136389 +0000 INFO cn-service.lockservice lock service start successfully {"uuid": "", "serviceID": "1715590458126536645s2"}
2024-05-13T09:15:25.7215074Z 2024/05/13 08:54:18.136651 +0000 INFO cn-service.lockservice.lock-client.lock-client gc idle backends task started {"uuid": ""}
2024-05-13T09:15:25.7215710Z 2024/05/13 08:54:18.136807 +0000 DEBUG cn-service.lockservice.lock-client.lock-client gc inactive backends task started {"uuid": ""}
2024-05-13T09:15:25.7216266Z 2024/05/13 08:54:18.136964 +0000 INFO cn-service.lockservice.allocator clean cannot commit task {"uuid": ""}
2024-05-13T09:15:25.7217108Z 2024/05/13 08:54:18.137837 +0000 DEBUG cn-service.lockservice.lock-server application accept loop started {"uuid": "", "name": "lock-server", "listen-addresses": "[/tmp/105227950.sock]"}
2024-05-13T09:15:25.7217945Z 2024/05/13 08:54:18.137952 +0000 DEBUG cn-service.lockservice.lock-server application accept loop stopped {"uuid": "", "name": "lock-server", "listen-addresses": "[/tmp/105227950.sock]"}
2024-05-13T09:15:25.7218726Z 2024/05/13 08:54:18.138015 +0000 DEBUG cn-service.lockservice.lock-server application started {"uuid": "", "name": "lock-server", "listen-addresses": "[/tmp/105227950.sock]"}
2024-05-13T09:15:25.7219208Z 2024/05/13 08:54:18.138069 +0000 INFO cn-service.lockservice lock allocator start successfully {"uuid": ""}
2024-05-13T09:15:25.7219748Z 2024/05/13 08:54:18.138160 +0000 DEBUG cn-service.lockservice txn created {"uuid": "", "txn": "74786e31()"}
2024-05-13T09:15:25.7220808Z 2024/05/13 08:54:18.138947 +0000 DEBUG cn-service.lockservice.lock-client start connect to remote {"uuid": "", "remote": "unix:///tmp/105227950.sock", "backend-id": "018f7128-eb1a-7486-b433-5c9ebfc17639"}
2024-05-13T09:15:25.7221581Z 2024/05/13 08:54:18.139182 +0000 DEBUG cn-service.lockservice.lock-server session init completed {"uuid": "", "name": "lock-server", "session-id": 1}
2024-05-13T09:15:25.7222515Z 2024/05/13 08:54:18.139298 +0000 DEBUG cn-service.lockservice.lock-server session connected {"uuid": "", "name": "lock-server", "listen-addresses": "[/tmp/105227950.sock]", "session-id": 1, "addr": "@"}
2024-05-13T09:15:25.7223501Z 2024/05/13 08:54:18.146061 +0000 DEBUG cn-service.lockservice.lock-client session init completed {"uuid": "", "remote": "unix:///tmp/105227950.sock", "backend-id": "018f7128-eb1a-7486-b433-5c9ebfc17639", "session-id": 0}
2024-05-13T09:15:25.7224437Z 2024/05/13 08:54:18.146118 +0000 DEBUG cn-service.lockservice.lock-client connect to remote succeed {"uuid": "", "remote": "unix:///tmp/105227950.sock", "backend-id": "018f7128-eb1a-7486-b433-5c9ebfc17639"}
2024-05-13T09:15:25.7225342Z 2024/05/13 08:54:18.146203 +0000 DEBUG cn-service.lockservice.lock-client read loop started {"uuid": "", "remote": "unix:///tmp/105227950.sock", "backend-id": "018f7128-eb1a-7486-b433-5c9ebfc17639"}
2024-05-13T09:15:25.7226257Z 2024/05/13 08:54:18.146336 +0000 DEBUG cn-service.lockservice.lock-client write loop started {"uuid": "", "remote": "unix:///tmp/105227950.sock", "backend-id": "018f7128-eb1a-7486-b433-5c9ebfc17639"}
2024-05-13T09:15:25.7227426Z 2024/05/13 08:54:18.146502 +0000 DEBUG cn-service.lockservice.lock-client write request {"uuid": "", "remote": "unix:///tmp/105227950.sock", "backend-id": "018f7128-eb1a-7486-b433-5c9ebfc17639", "request-id": 1, "request": "1: GetBind/1715590458105442711s1-0"}
2024-05-13T09:15:25.7228492Z 2024/05/13 08:54:18.146706 +0000 DEBUG cn-service.lockservice.lock-server session read message {"uuid": "", "name": "lock-server", "listen-addresses": "[/tmp/105227950.sock]", "session-id": 1, "addr": "@", "seqence": 1}
2024-05-13T09:15:25.7229456Z 2024/05/13 08:54:18.146810 +0000 DEBUG cn-service.lockservice.lock-server received request {"uuid": "", "name": "lock-server", "sequence": 1, "client": "@", "request-id": 1, "request": "1: GetBind/1715590458105442711s1-0"}
2024-05-13T09:15:25.7230080Z 2024/05/13 08:54:18.146929 +0000 DEBUG cn-service.lockservice received a request {"uuid": "", "request": "1: GetBind/1715590458105442711s1-0"}
2024-05-13T09:15:25.7230896Z 2024/05/13 08:54:18.146989 +0000 DEBUG cn-service.lockservice.lock-server handle request completed {"uuid": "", "name": "lock-server", "sequence": 1, "client": "@", "request-id": 1}
2024-05-13T09:15:25.7231550Z 2024/05/13 08:54:18.148335 +0000 INFO cn-service.lockservice.allocator bind created {"uuid": "", "table": 0, "service": "1715590458105442711s1"}
2024-05-13T09:15:25.7232217Z 2024/05/13 08:54:18.148430 +0000 DEBUG cn-service.lockservice handle request completed {"uuid": "", "response": "1: GetBind/0-1715590458105442711s1-1"}
2024-05-13T09:15:25.7233198Z 2024/05/13 08:54:18.154120 +0000 DEBUG cn-service.lockservice.lock-server write responses {"uuid": "", "name": "lock-server", "client": "@", "request-id": 1, "response": "1: GetBind/0-1715590458105442711s1-1"}
2024-05-13T09:15:25.7234374Z 2024/05/13 08:54:18.154329 +0000 DEBUG cn-service.lockservice.lock-client read response {"uuid": "", "remote": "unix:///tmp/105227950.sock", "backend-id": "018f7128-eb1a-7486-b433-5c9ebfc17639", "request-id": 1, "response": "1: GetBind/0-1715590458105442711s1-1"}
2024-05-13T09:15:25.7235137Z 2024/05/13 08:54:18.160351 +0000 INFO cn-service.lockservice bind created {"uuid": "", "service": "1715590458105442711s1", "remote": false, "bind": "0-1715590458105442711s1-1"}
2024-05-13T09:15:25.7235956Z 2024/05/13 08:54:18.160462 +0000 DEBUG cn-service.lockservice try to lock on local {"uuid": "", "txn": "74786e31()", "table": 0, "rows": "[01]", "opts": "Exclusive-Row-Wait"}
2024-05-13T09:15:25.7236915Z 2024/05/13 08:54:18.160565 +0000 DEBUG cn-service.lockservice holder added {"uuid": "", "txn": "74786e31()", "table": 0, "rows": "[01]", "opts": "Exclusive-Row-Wait", "holders": "[74786e31(1715590458105442711s1)]", "waiters": "[]"}
2024-05-13T09:15:25.7237549Z 2024/05/13 08:54:18.160657 +0000 DEBUG cn-service.lockservice lock added to txn {"uuid": "", "txn": "74786e31()", "rows": "[01]"}2024-05-13T09:15:25.7238277Z 2024/05/13 08:54:18.160745 +0000 DEBUG cn-service.lockservice lock added to local {"uuid": "", "txn": "74786e31()", "table": 0, "rows": "[01]", "opts": "Exclusive-Row-Wait"}
2024-05-13T09:15:25.7238757Z 2024/05/13 08:54:18.160834 +0000 DEBUG cn-service.lockservice unlock txn {"uuid": "", "txn": "74786e31()"}
2024-05-13T09:15:25.7239264Z 2024/05/13 08:54:18.160885 +0000 DEBUG cn-service.lockservice ready to unlock txn {"uuid": "", "txn": "74786e31()"}
2024-05-13T09:15:25.7239828Z 2024/05/13 08:54:18.160939 +0000 DEBUG cn-service.lockservice txn unlock table {"uuid": "", "txn": "74786e31()", "table": 0}
2024-05-13T09:15:25.7240508Z 2024/05/13 08:54:18.160995 +0000 DEBUG cn-service.lockservice txn unlock table on local {"uuid": "", "txn": "74786e31()", "bind": "0-1715590458105442711s1-1"}
2024-05-13T09:15:25.7241215Z 2024/05/13 08:54:18.161064 +0000 DEBUG cn-service.lockservice lock unlocked {"uuid": "", "txn": "74786e31()", "key": "01", "lock": "Exclusive(row): holder(), waiters()"}
2024-05-13T09:15:25.7241883Z 2024/05/13 08:54:18.161153 +0000 DEBUG cn-service.lockservice txn unlock table completed {"uuid": "", "txn": "74786e31()", "table": 0, "rows": "[01]"}
2024-05-13T09:15:25.7242440Z 2024/05/13 08:54:18.161217 +0000 DEBUG cn-service.lockservice unlock txn {"uuid": "", "txn": "74786e31()", "cost": "335.887µs"}
2024-05-13T09:15:25.7242913Z 2024/05/13 08:54:18.161299 +0000 DEBUG cn-service.lockservice txn created {"uuid": "", "txn": "74786e32()"}
2024-05-13T09:15:25.7243927Z 2024/05/13 08:54:18.161719 +0000 DEBUG cn-service.lockservice.lock-client start connect to remote {"uuid": "", "remote": "unix:///tmp/105227950.sock", "backend-id": "018f7128-eb31-7698-9a78-1b6b1c5c0193"}
2024-05-13T09:15:25.7244266Z panic: missing free for type: *lockservice.waiter, 03-0xc0000d6780(1)
2024-05-13T09:15:25.7244383Z  create by: <<<>>>
2024-05-13T09:15:25.7244390Z
2024-05-13T09:15:25.7244396Z
2024-05-13T09:15:25.7244516Z goroutine 5 [running]:
2024-05-13T09:15:25.7244917Z github.com/matrixorigin/matrixone/pkg/common/reuse.(*checker[...]).gc(0x1215c40, 0xc0000d6780)
2024-05-13T09:15:25.7245289Z    /home/runner/work/matrixone/matrixone/pkg/common/reuse/checker.go:133 +0x394
2024-05-13T09:15:25.7245645Z github.com/matrixorigin/matrixone/pkg/common/reuse.newSyncPoolBased[...].func1.1()
2024-05-13T09:15:25.7246020Z    /home/runner/work/matrixone/matrixone/pkg/common/reuse/sync_pool_based.go:46 +0xc5
2024-05-13T09:15:25.7246283Z FAIL   github.com/matrixorigin/matrixone/pkg/lockservice   190.856s
2024-05-13T09:15:25.7247079Z 2024/05/13 08:51:09.675567 +0000 INFO malloc/malloc.go:42 malloc {"max buffer size": 1073741824, "num shards": 16, "classes": 23, "min class size": 128, "max class size": 1048576, "buffer objects per class": 23}
2024-05-13T09:15:25.7247588Z 2024/05/13 08:51:09.814962 +0000 INFO logutil/internal.go:37 MO logger init, level=debug, log file=, stackLevel=
2024-05-13T09:15:25.7247735Z === RUN   TestClientConfigIsValidated

@daviszhen
Copy link
Contributor

daviszhen commented May 15, 2024 •
edited
Loading

https://github.com/matrixorigin/matrixone/actions/runs/9094783677/job/24996592593?pr=16147

2024-05-15T11:46:30.2714693Z 2024/05/15 11:32:20.269424 +0000 DEBUG cn-service.lockservice lock added to txn {"uuid": "", "txn": "74786e31()", "rows": "[01]"}
2024-05-15T11:46:30.2715437Z 2024/05/15 11:32:20.269495 +0000 DEBUG cn-service.lockservice lock added to local {"uuid": "", "txn": "74786e31()", "table": 0, "rows": "[01]", "opts": "Exclusive-Row-Wait"}
2024-05-15T11:46:30.2715995Z 2024/05/15 11:32:20.269578 +0000 INFO cn-service.lockservice keep remote locks task {"uuid": "", "cost": "12.526836ms"}
2024-05-15T11:46:30.2716553Z 2024/05/15 11:32:20.269624 +0000 INFO cn-service.lockservice keep lock table bind task {"uuid": "", "cost": "12.591277ms"}
2024-05-15T11:46:30.2716886Z panic: missing free for type: *lockservice.waiter, 02-0xc000264300(1)
2024-05-15T11:46:30.2717007Z create by: <<<>>>
2024-05-15T11:46:30.2717015Z
2024-05-15T11:46:30.2717019Z
2024-05-15T11:46:30.2717134Z goroutine 18 [running]:
2024-05-15T11:46:30.2717544Z github.com/matrixorigin/matrixone/pkg/common/reuse.(*checker[...]).gc(0x1216e80, 0xc000264300)
2024-05-15T11:46:30.2717915Z /home/runner/work/matrixone/matrixone/pkg/common/reuse/checker.go:133 +0x394
2024-05-15T11:46:30.2718276Z github.com/matrixorigin/matrixone/pkg/common/reuse.newSyncPoolBased[...].func1.1()
2024-05-15T11:46:30.2718656Z /home/runner/work/matrixone/matrixone/pkg/common/reuse/sync_pool_based.go:46 +0xc5
2024-05-15T11:46:30.2718990Z FAIL github.com/matrixorigin/matrixone/pkg/lockservice 194.090s
2024-05-15T11:46:30.2719807Z 2024/05/15 11:29:12.761045 +0000 INFO malloc/malloc.go:42 malloc {"max buffer size": 1073741824, "num shards": 16, "classes": 23, "min class size": 128, "max class size": 1048576, "buffer objects per class": 23}
2024-05-15T11:46:30.2720258Z 2024/05/15 11:29:12.926658 +0000 INFO logutil/internal.go:37 MO logger init, level=debug, log file=, stackLevel=

This was referenced May 16, 2024
Merged
Merged
@ck89119
Copy link
Contributor

ck89119 commented May 27, 2024 •
edited
Loading

https://github.com/matrixorigin/matrixone/actions/runs/9248603496/attempts/1?pr=16408

2024-05-27T03:47:09.0074913Z === RUN   TestLockSuccWithKeepBindTimeout
2024-05-27T03:47:09.0075605Z 2024/05/27 03:18:19.310819 +0000 INFO cn-service.lockservice dead lock checker {"uuid": ""}
2024-05-27T03:47:09.0076464Z 2024/05/27 03:18:19.310819 +0000 INFO cn-service.lockservice dead lock checker {"uuid": ""}
2024-05-27T03:47:09.0077305Z 2024/05/27 03:18:19.310899 +0000 INFO cn-service.lockservice dead lock checker {"uuid": ""}
2024-05-27T03:47:09.0078156Z 2024/05/27 03:18:19.311008 +0000 INFO cn-service.lockservice dead lock checker {"uuid": ""}
2024-05-27T03:47:09.0079217Z 2024/05/27 03:18:19.311024 +0000 INFO cn-service.lockservice.lock-client.lock-client gc idle backends task started {"uuid": ""}
2024-05-27T03:47:09.0080508Z 2024/05/27 03:18:19.311123 +0000 DEBUG cn-service.lockservice.lock-client.lock-client gc inactive backends task started {"uuid": ""}
2024-05-27T03:47:09.0081600Z 2024/05/27 03:18:19.312664 +0000 INFO cn-service.lockservice keep lock table bind task {"uuid": ""}
2024-05-27T03:47:09.0082519Z 2024/05/27 03:18:19.312666 +0000 INFO cn-service.lockservice keep remote locks task {"uuid": ""}
2024-05-27T03:47:09.0083899Z 2024/05/27 03:18:19.313229 +0000 DEBUG cn-service.lockservice.lock-server application accept loop started {"uuid": "", "name": "lock-server", "listen-addresses": "[/tmp/service-308974149-s1.sock]"}
2024-05-27T03:47:09.0085794Z 2024/05/27 03:18:19.313296 +0000 DEBUG cn-service.lockservice.lock-server application accept loop stopped {"uuid": "", "name": "lock-server", "listen-addresses": "[/tmp/service-308974149-s1.sock]"}
2024-05-27T03:47:09.0087564Z 2024/05/27 03:18:19.313339 +0000 DEBUG cn-service.lockservice.lock-server application started {"uuid": "", "name": "lock-server", "listen-addresses": "[/tmp/service-308974149-s1.sock]"}
2024-05-27T03:47:09.0089019Z 2024/05/27 03:18:19.313556 +0000 INFO cn-service.lockservice lock service start successfully {"uuid": "", "serviceID": "1716779899309059057s1"}
2024-05-27T03:47:09.0090281Z 2024/05/27 03:18:19.317861 +0000 INFO cn-service.lockservice.lock-client.lock-client gc idle backends task started {"uuid": ""}
2024-05-27T03:47:09.0091543Z 2024/05/27 03:18:19.317939 +0000 DEBUG cn-service.lockservice.lock-client.lock-client gc inactive backends task started {"uuid": ""}
2024-05-27T03:47:09.0092682Z 2024/05/27 03:18:19.318041 +0000 INFO cn-service.lockservice.allocator clean cannot commit task {"uuid": ""}
2024-05-27T03:47:09.0094063Z 2024/05/27 03:18:19.318634 +0000 DEBUG cn-service.lockservice.lock-server application accept loop started {"uuid": "", "name": "lock-server", "listen-addresses": "[/tmp/308825642.sock]"}
2024-05-27T03:47:09.0095703Z 2024/05/27 03:18:19.318697 +0000 DEBUG cn-service.lockservice.lock-server application accept loop stopped {"uuid": "", "name": "lock-server", "listen-addresses": "[/tmp/308825642.sock]"}
2024-05-27T03:47:09.0097296Z 2024/05/27 03:18:19.318738 +0000 DEBUG cn-service.lockservice.lock-server application started {"uuid": "", "name": "lock-server", "listen-addresses": "[/tmp/308825642.sock]"}
2024-05-27T03:47:09.0098544Z 2024/05/27 03:18:19.318779 +0000 INFO cn-service.lockservice lock allocator start successfully {"uuid": ""}
2024-05-27T03:47:09.0099280Z panic: test timed out after 30m0s
2024-05-27T03:47:09.0099628Z running tests:
2024-05-27T03:47:09.0099964Z 	TestLockSuccWithKeepBindTimeout (26m32s)
2024-05-27T03:47:09.0100259Z 
2024-05-27T03:47:09.0100379Z goroutine 62860 [running]:
2024-05-27T03:47:09.0100708Z testing.(*M).startAlarm.func1()
2024-05-27T03:47:09.0101252Z 	/opt/hostedtoolcache/go/1.21.5/x64/src/testing/testing.go:2259 +0x259
2024-05-27T03:47:09.0101779Z created by time.goFunc
2024-05-27T03:47:09.0102233Z 	/opt/hostedtoolcache/go/1.21.5/x64/src/time/sleep.go:176 +0x45
2024-05-27T03:47:09.0102592Z 
2024-05-27T03:47:09.0102737Z goroutine 1 [chan receive, 26 minutes]:
2024-05-27T03:47:09.0103209Z testing.(*T).Run(0xc00009a9c0, {0x106f1db, 0x1f}, 0x10a8850)
2024-05-27T03:47:09.0103829Z 	/opt/hostedtoolcache/go/1.21.5/x64/src/testing/testing.go:1649 +0x871
2024-05-27T03:47:09.0104342Z testing.runTests.func1(0x0?)
2024-05-27T03:47:09.0104836Z 	/opt/hostedtoolcache/go/1.21.5/x64/src/testing/testing.go:2054 +0x85
2024-05-27T03:47:09.0105477Z testing.tRunner(0xc00009a9c0, 0xc0001ffaf8)
2024-05-27T03:47:09.0106026Z 	/opt/hostedtoolcache/go/1.21.5/x64/src/testing/testing.go:1595 +0x262
2024-05-27T03:47:09.0106717Z testing.runTests(0xc0000ee000?, {0x17c2f80, 0xb9, 0xb9}, {0x1c?, 0x1e?, 0x17c9340?})
2024-05-27T03:47:09.0107424Z 	/opt/hostedtoolcache/go/1.21.5/x64/src/testing/testing.go:2052 +0x8ae
2024-05-27T03:47:09.0107923Z testing.(*M).Run(0xc0000ee000)
2024-05-27T03:47:09.0108408Z 	/opt/hostedtoolcache/go/1.21.5/x64/src/testing/testing.go:1925 +0xcd8
2024-05-27T03:47:09.0108879Z main.main()
2024-05-27T03:47:09.0109156Z 	_testmain.go:427 +0x2be
2024-05-27T03:47:09.0109360Z 
2024-05-27T03:47:09.0109471Z goroutine 6 [select]:
2024-05-27T03:47:09.0110042Z github.com/panjf2000/ants/v2.(*Pool).purgeStaleWorkers(0xc000112f00, {0x1222fd0, 0xc0001507d0})
2024-05-27T03:47:09.0110866Z 	/home/runner/go/pkg/mod/github.com/panjf2000/ants/v2@v2.7.4/pool.go:83 +0x1eb
2024-05-27T03:47:09.0111577Z created by github.com/panjf2000/ants/v2.(*Pool).goPurge in goroutine 1
2024-05-27T03:47:09.0112308Z 	/home/runner/go/pkg/mod/github.com/panjf2000/ants/v2@v2.7.4/pool.go:148 +0x185
2024-05-27T03:47:09.0112742Z 
2024-05-27T03:47:09.0112872Z goroutine 7 [select]:
2024-05-27T03:47:09.0113530Z github.com/panjf2000/ants/v2.(*Pool).ticktock(0xc000112f00, {0x1222fd0, 0xc000150820})
2024-05-27T03:47:09.0114355Z 	/home/runner/go/pkg/mod/github.com/panjf2000/ants/v2@v2.7.4/pool.go:126 +0x205
2024-05-27T03:47:09.0115161Z created by github.com/panjf2000/ants/v2.(*Pool).goTicktock in goroutine 1
2024-05-27T03:47:09.0115889Z 	/home/runner/go/pkg/mod/github.com/panjf2000/ants/v2@v2.7.4/pool.go:155 +0x187
2024-05-27T03:47:09.0116308Z 
2024-05-27T03:47:09.0116433Z goroutine 116 [runnable]:
2024-05-27T03:47:09.0116744Z runtime.Gosched(...)
2024-05-27T03:47:09.0117165Z 	/opt/hostedtoolcache/go/1.21.5/x64/src/runtime/proc.go:338
2024-05-27T03:47:09.0117604Z runtime.GC()
2024-05-27T03:47:09.0118012Z 	/opt/hostedtoolcache/go/1.21.5/x64/src/runtime/mgc.go:477 +0x65
2024-05-27T03:47:09.0118498Z runtime/debug.freeOSMemory()
2024-05-27T03:47:09.0118984Z 	/opt/hostedtoolcache/go/1.21.5/x64/src/runtime/mheap.go:1672 +0x13
2024-05-27T03:47:09.0119484Z runtime/debug.FreeOSMemory(...)
2024-05-27T03:47:09.0120000Z 	/opt/hostedtoolcache/go/1.21.5/x64/src/runtime/debug/garbage.go:104
2024-05-27T03:47:09.0120649Z github.com/matrixorigin/matrixone/pkg/common/mpool.init.0.func1()
2024-05-27T03:47:09.0121397Z 	/home/runner/work/matrixone/matrixone/pkg/common/mpool/alloc_tracing.go:35 +0x4a
2024-05-27T03:47:09.0122197Z created by github.com/matrixorigin/matrixone/pkg/common/mpool.init.0 in goroutine 1
2024-05-27T03:47:09.0123015Z 	/home/runner/work/matrixone/matrixone/pkg/common/mpool/alloc_tracing.go:31 +0x25
2024-05-27T03:47:09.0123472Z 
2024-05-27T03:47:09.0123614Z goroutine 50011 [select, 26 minutes]:
2024-05-27T03:47:09.0124366Z github.com/matrixorigin/matrixone/pkg/common/morpc.(*client).gcInactiveTask(0xc0000debb0, {0x1222f98, 0xc0005b4000})
2024-05-27T03:47:09.0125420Z 	/home/runner/work/matrixone/matrixone/pkg/common/morpc/client.go:406 +0x1ff
2024-05-27T03:47:09.0126260Z github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask.func1()
2024-05-27T03:47:09.0127127Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:277 +0xe8
2024-05-27T03:47:09.0128068Z created by github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask in goroutine 49972
2024-05-27T03:47:09.0129029Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:272 +0x167
2024-05-27T03:47:09.0129482Z 
2024-05-27T03:47:09.0129629Z goroutine 49999 [IO wait, 26 minutes]:
2024-05-27T03:47:09.0130081Z internal/poll.runtime_pollWait(0x7f259adc3ff8, 0x72)
2024-05-27T03:47:09.0130683Z 	/opt/hostedtoolcache/go/1.21.5/x64/src/runtime/netpoll.go:343 +0x85
2024-05-27T03:47:09.0131266Z internal/poll.(*pollDesc).wait(0xc00295ec20, 0x1?, 0x0)
2024-05-27T03:47:09.0131923Z 	/opt/hostedtoolcache/go/1.21.5/x64/src/internal/poll/fd_poll_runtime.go:84 +0xb1
2024-05-27T03:47:09.0132502Z internal/poll.(*pollDesc).waitRead(...)
2024-05-27T03:47:09.0133068Z 	/opt/hostedtoolcache/go/1.21.5/x64/src/internal/poll/fd_poll_runtime.go:89
2024-05-27T03:47:09.0133618Z internal/poll.(*FD).Accept(0xc00295ec00)
2024-05-27T03:47:09.0134184Z 	/opt/hostedtoolcache/go/1.21.5/x64/src/internal/poll/fd_unix.go:611 +0x425
2024-05-27T03:47:09.0134709Z net.(*netFD).accept(0xc00295ec00)
2024-05-27T03:47:09.0135192Z 	/opt/hostedtoolcache/go/1.21.5/x64/src/net/fd_unix.go:172 +0x3e
2024-05-27T03:47:09.0135690Z net.(*UnixListener).accept(0xc0003c4cc0)
2024-05-27T03:47:09.0136231Z 	/opt/hostedtoolcache/go/1.21.5/x64/src/net/unixsock_posix.go:172 +0x3e
2024-05-27T03:47:09.0136757Z net.(*UnixListener).Accept(0xc0003c4cc0)
2024-05-27T03:47:09.0137264Z 	/opt/hostedtoolcache/go/1.21.5/x64/src/net/unixsock.go:260 +0x65
2024-05-27T03:47:09.0137583Z github.com/fagongzi/goetty/v2.(*server).doStart.func2({0x1220c70, 0xc0003c4cc0})
2024-05-27T03:47:09.0138232Z 	/home/runner/go/pkg/mod/github.com/matrixorigin/goetty/v2@v2.0.0-20231122095211-6a25dc9130ca/application.go:309 +0xf0
2024-05-27T03:47:09.0138544Z created by github.com/fagongzi/goetty/v2.(*server).doStart in goroutine 49972
2024-05-27T03:47:09.0139371Z 	/home/runner/go/pkg/mod/github.com/matrixorigin/goetty/v2@v2.0.0-20231122095211-6a25dc9130ca/application.go:365 +0x187
2024-05-27T03:47:09.0139379Z 
2024-05-27T03:47:09.0139530Z goroutine 50020 [select, 26 minutes]:
2024-05-27T03:47:09.0140063Z github.com/matrixorigin/matrixone/pkg/lockservice.(*server).handle(0xc003b8f2c0, {0x1222f98, 0xc0005b5cb0})
2024-05-27T03:47:09.0140396Z 	/home/runner/work/matrixone/matrixone/pkg/lockservice/rpc.go:375 +0x1e7
2024-05-27T03:47:09.0140798Z github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask.func1()
2024-05-27T03:47:09.0141151Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:277 +0xe8
2024-05-27T03:47:09.0141647Z created by github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask in goroutine 49972
2024-05-27T03:47:09.0142012Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:272 +0x167
2024-05-27T03:47:09.0142018Z 
2024-05-27T03:47:09.0142135Z goroutine 49978 [select]:
2024-05-27T03:47:09.0142633Z github.com/matrixorigin/matrixone/pkg/common/morpc.(*client).gcIdleTask(0xc0000de9a0, {0x1222f98, 0xc0003c49c0})
2024-05-27T03:47:09.0142972Z 	/home/runner/work/matrixone/matrixone/pkg/common/morpc/client.go:383 +0x292
2024-05-27T03:47:09.0143372Z github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask.func1()
2024-05-27T03:47:09.0143723Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:277 +0xe8
2024-05-27T03:47:09.0144229Z created by github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask in goroutine 49972
2024-05-27T03:47:09.0144583Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:272 +0x167
2024-05-27T03:47:09.0144651Z 
2024-05-27T03:47:09.0144772Z goroutine 50001 [select]:
2024-05-27T03:47:09.0145271Z github.com/matrixorigin/matrixone/pkg/lockservice.(*waiterEvents).handle(0xc004086bd0, {0x1222f98, 0xc00010faa0})
2024-05-27T03:47:09.0145761Z 	/home/runner/work/matrixone/matrixone/pkg/lockservice/waiter_events.go:183 +0x2ba
2024-05-27T03:47:09.0146172Z github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask.func1()
2024-05-27T03:47:09.0146523Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:277 +0xe8
2024-05-27T03:47:09.0147017Z created by github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask in goroutine 49972
2024-05-27T03:47:09.0147366Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:272 +0x167
2024-05-27T03:47:09.0147372Z 
2024-05-27T03:47:09.0147511Z goroutine 50006 [select, 26 minutes]:
2024-05-27T03:47:09.0148052Z github.com/matrixorigin/matrixone/pkg/lockservice.(*service).handleFetchWhoWaitingMe(0xc000152780, {0x1222f98, 0xc00010fc80})
2024-05-27T03:47:09.0148431Z 	/home/runner/work/matrixone/matrixone/pkg/lockservice/service_remote.go:546 +0x13a
2024-05-27T03:47:09.0148827Z github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask.func1()
2024-05-27T03:47:09.0149174Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:277 +0xe8
2024-05-27T03:47:09.0149683Z created by github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask in goroutine 49972
2024-05-27T03:47:09.0150063Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:272 +0x167
2024-05-27T03:47:09.0150069Z 
2024-05-27T03:47:09.0150212Z goroutine 49986 [select, 26 minutes]:
2024-05-27T03:47:09.0150671Z github.com/matrixorigin/matrixone/pkg/lockservice.(*server).handle(0xc003b8e4b0, {0x1222f98, 0xc0002af710})
2024-05-27T03:47:09.0150997Z 	/home/runner/work/matrixone/matrixone/pkg/lockservice/rpc.go:375 +0x1e7
2024-05-27T03:47:09.0151390Z github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask.func1()
2024-05-27T03:47:09.0151743Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:277 +0xe8
2024-05-27T03:47:09.0152247Z created by github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask in goroutine 49972
2024-05-27T03:47:09.0152746Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:272 +0x167
2024-05-27T03:47:09.0152753Z 
2024-05-27T03:47:09.0152953Z goroutine 50026 [select, 26 minutes]:
2024-05-27T03:47:09.0153420Z github.com/matrixorigin/matrixone/pkg/lockservice.(*server).handle(0xc003b8f2c0, {0x1222f98, 0xc0005b5ef0})
2024-05-27T03:47:09.0153752Z 	/home/runner/work/matrixone/matrixone/pkg/lockservice/rpc.go:375 +0x1e7
2024-05-27T03:47:09.0154142Z github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask.func1()
2024-05-27T03:47:09.0154490Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:277 +0xe8
2024-05-27T03:47:09.0154986Z created by github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask in goroutine 49972
2024-05-27T03:47:09.0155344Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:272 +0x167
2024-05-27T03:47:09.0155350Z 
2024-05-27T03:47:09.0155484Z goroutine 49989 [select, 26 minutes]:
2024-05-27T03:47:09.0155953Z github.com/matrixorigin/matrixone/pkg/lockservice.(*server).handle(0xc003b8e4b0, {0x1222f98, 0xc0002af890})
2024-05-27T03:47:09.0156268Z 	/home/runner/work/matrixone/matrixone/pkg/lockservice/rpc.go:375 +0x1e7
2024-05-27T03:47:09.0156673Z github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask.func1()
2024-05-27T03:47:09.0157013Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:277 +0xe8
2024-05-27T03:47:09.0157513Z created by github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask in goroutine 49972
2024-05-27T03:47:09.0157858Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:272 +0x167
2024-05-27T03:47:09.0157943Z 
2024-05-27T03:47:09.0158128Z goroutine 49974 [select, 26 minutes]:
2024-05-27T03:47:09.0158604Z github.com/matrixorigin/matrixone/pkg/lockservice.(*detector).doCheck(0xc004df2480, {0x1222f98, 0xc0003c46f0})
2024-05-27T03:47:09.0158951Z 	/home/runner/work/matrixone/matrixone/pkg/lockservice/deadlock.go:123 +0x29d
2024-05-27T03:47:09.0159345Z github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask.func1()
2024-05-27T03:47:09.0159690Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:277 +0xe8
2024-05-27T03:47:09.0160183Z created by github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask in goroutine 49972
2024-05-27T03:47:09.0160525Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:272 +0x167
2024-05-27T03:47:09.0160530Z 
2024-05-27T03:47:09.0160667Z goroutine 50015 [select, 26 minutes]:
2024-05-27T03:47:09.0161111Z github.com/matrixorigin/matrixone/pkg/lockservice.(*server).handle(0xc003b8f2c0, {0x1222f98, 0xc0005b5500})
2024-05-27T03:47:09.0161437Z 	/home/runner/work/matrixone/matrixone/pkg/lockservice/rpc.go:375 +0x1e7
2024-05-27T03:47:09.0161824Z github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask.func1()
2024-05-27T03:47:09.0162190Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:277 +0xe8
2024-05-27T03:47:09.0162690Z created by github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask in goroutine 49972
2024-05-27T03:47:09.0163032Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:272 +0x167
2024-05-27T03:47:09.0163037Z 
2024-05-27T03:47:09.0163174Z goroutine 49993 [select, 26 minutes]:
2024-05-27T03:47:09.0163612Z github.com/matrixorigin/matrixone/pkg/lockservice.(*server).handle(0xc003b8e4b0, {0x1222f98, 0xc0002afa40})
2024-05-27T03:47:09.0163932Z 	/home/runner/work/matrixone/matrixone/pkg/lockservice/rpc.go:375 +0x1e7
2024-05-27T03:47:09.0164315Z github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask.func1()
2024-05-27T03:47:09.0164663Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:277 +0xe8
2024-05-27T03:47:09.0165147Z created by github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask in goroutine 49972
2024-05-27T03:47:09.0165611Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:272 +0x167
2024-05-27T03:47:09.0165616Z 
2024-05-27T03:47:09.0165807Z goroutine 50028 [select, 26 minutes]:
2024-05-27T03:47:09.0166264Z github.com/matrixorigin/matrixone/pkg/lockservice.(*server).handle(0xc003b8f2c0, {0x1222f98, 0xc0005b5fb0})
2024-05-27T03:47:09.0166581Z 	/home/runner/work/matrixone/matrixone/pkg/lockservice/rpc.go:375 +0x1e7
2024-05-27T03:47:09.0166975Z github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask.func1()
2024-05-27T03:47:09.0167319Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:277 +0xe8
2024-05-27T03:47:09.0167808Z created by github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask in goroutine 49972
2024-05-27T03:47:09.0168158Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:272 +0x167
2024-05-27T03:47:09.0168163Z 
2024-05-27T03:47:09.0168293Z goroutine 49992 [select, 26 minutes]:
2024-05-27T03:47:09.0168740Z github.com/matrixorigin/matrixone/pkg/lockservice.(*server).handle(0xc003b8e4b0, {0x1222f98, 0xc0002af9e0})
2024-05-27T03:47:09.0169052Z 	/home/runner/work/matrixone/matrixone/pkg/lockservice/rpc.go:375 +0x1e7
2024-05-27T03:47:09.0169443Z github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask.func1()
2024-05-27T03:47:09.0169778Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:277 +0xe8
2024-05-27T03:47:09.0170272Z created by github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask in goroutine 49972
2024-05-27T03:47:09.0170610Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:272 +0x167
2024-05-27T03:47:09.0170676Z 
2024-05-27T03:47:09.0170803Z goroutine 50003 [select]:
2024-05-27T03:47:09.0171282Z github.com/matrixorigin/matrixone/pkg/lockservice.(*waiterEvents).handle(0xc004086bd0, {0x1222f98, 0xc00010fb60})
2024-05-27T03:47:09.0171658Z 	/home/runner/work/matrixone/matrixone/pkg/lockservice/waiter_events.go:183 +0x2ba
2024-05-27T03:47:09.0172070Z github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask.func1()
2024-05-27T03:47:09.0172424Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:277 +0xe8
2024-05-27T03:47:09.0172929Z created by github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask in goroutine 49972
2024-05-27T03:47:09.0173281Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:272 +0x167
2024-05-27T03:47:09.0173286Z 
2024-05-27T03:47:09.0173426Z goroutine 49979 [select, 26 minutes]:
2024-05-27T03:47:09.0173925Z github.com/matrixorigin/matrixone/pkg/common/morpc.(*client).gcInactiveTask(0xc0000de9a0, {0x1222f98, 0xc0003c4a20})
2024-05-27T03:47:09.0174266Z 	/home/runner/work/matrixone/matrixone/pkg/common/morpc/client.go:406 +0x1ff
2024-05-27T03:47:09.0174650Z github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask.func1()
2024-05-27T03:47:09.0175000Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:277 +0xe8
2024-05-27T03:47:09.0175496Z created by github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask in goroutine 49972
2024-05-27T03:47:09.0175847Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:272 +0x167
2024-05-27T03:47:09.0175852Z 
2024-05-27T03:47:09.0175983Z goroutine 49997 [select, 26 minutes]:
2024-05-27T03:47:09.0176429Z github.com/matrixorigin/matrixone/pkg/lockservice.(*server).handle(0xc003b8e4b0, {0x1222f98, 0xc0002afbc0})
2024-05-27T03:47:09.0176752Z 	/home/runner/work/matrixone/matrixone/pkg/lockservice/rpc.go:375 +0x1e7
2024-05-27T03:47:09.0177137Z github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask.func1()
2024-05-27T03:47:09.0177488Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:277 +0xe8
2024-05-27T03:47:09.0177981Z created by github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask in goroutine 49972
2024-05-27T03:47:09.0178444Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:272 +0x167
2024-05-27T03:47:09.0178451Z 
2024-05-27T03:47:09.0178636Z goroutine 49973 [select, 26 minutes]:
2024-05-27T03:47:09.0179108Z github.com/matrixorigin/matrixone/pkg/lockservice.(*detector).doCheck(0xc004df2480, {0x1222f98, 0xc0003c4690})
2024-05-27T03:47:09.0179451Z 	/home/runner/work/matrixone/matrixone/pkg/lockservice/deadlock.go:123 +0x29d
2024-05-27T03:47:09.0179843Z github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask.func1()
2024-05-27T03:47:09.0180180Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:277 +0xe8
2024-05-27T03:47:09.0180676Z created by github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask in goroutine 49972
2024-05-27T03:47:09.0181018Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:272 +0x167
2024-05-27T03:47:09.0181029Z 
2024-05-27T03:47:09.0181145Z goroutine 50002 [select]:
2024-05-27T03:47:09.0181626Z github.com/matrixorigin/matrixone/pkg/lockservice.(*waiterEvents).handle(0xc004086bd0, {0x1222f98, 0xc00010fb00})
2024-05-27T03:47:09.0181998Z 	/home/runner/work/matrixone/matrixone/pkg/lockservice/waiter_events.go:183 +0x2ba
2024-05-27T03:47:09.0182398Z github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask.func1()
2024-05-27T03:47:09.0182744Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:277 +0xe8
2024-05-27T03:47:09.0183235Z created by github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask in goroutine 49972
2024-05-27T03:47:09.0183585Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:272 +0x167
2024-05-27T03:47:09.0183651Z 
2024-05-27T03:47:09.0183794Z goroutine 50008 [select, 26 minutes]:
2024-05-27T03:47:09.0184334Z github.com/matrixorigin/matrixone/pkg/lockservice.(*service).handleFetchWhoWaitingMe(0xc000152780, {0x1222f98, 0xc00010fd40})
2024-05-27T03:47:09.0184712Z 	/home/runner/work/matrixone/matrixone/pkg/lockservice/service_remote.go:546 +0x13a
2024-05-27T03:47:09.0185096Z github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask.func1()
2024-05-27T03:47:09.0185583Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:277 +0xe8
2024-05-27T03:47:09.0186085Z created by github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask in goroutine 49972
2024-05-27T03:47:09.0186428Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:272 +0x167
2024-05-27T03:47:09.0186433Z 
2024-05-27T03:47:09.0186572Z goroutine 50009 [select, 26 minutes]:
2024-05-27T03:47:09.0187050Z github.com/matrixorigin/matrixone/pkg/common/morpc.(*client).createTask(0xc0000debb0, {0x1222f98, 0xc00010ff20})
2024-05-27T03:47:09.0187396Z 	/home/runner/work/matrixone/matrixone/pkg/common/morpc/client.go:458 +0x113
2024-05-27T03:47:09.0187779Z github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask.func1()
2024-05-27T03:47:09.0188123Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:277 +0xe8
2024-05-27T03:47:09.0188607Z created by github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask in goroutine 49972
2024-05-27T03:47:09.0188958Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:272 +0x167
2024-05-27T03:47:09.0188963Z 
2024-05-27T03:47:09.0189077Z goroutine 50012 [select]:
2024-05-27T03:47:09.0189652Z github.com/matrixorigin/matrixone/pkg/lockservice.(*lockTableAllocator).checkInvalidBinds(0xc0002a8be0, {0x1222f98, 0xc0005b4390})
2024-05-27T03:47:09.0190052Z 	/home/runner/work/matrixone/matrixone/pkg/lockservice/lock_table_allocator.go:441 +0x1cb
2024-05-27T03:47:09.0190462Z github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask.func1()
2024-05-27T03:47:09.0190813Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:277 +0xe8
2024-05-27T03:47:09.0191473Z created by github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask in goroutine 49972
2024-05-27T03:47:09.0191838Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:272 +0x167
2024-05-27T03:47:09.0191908Z 
2024-05-27T03:47:09.0192042Z goroutine 49994 [select, 26 minutes]:
2024-05-27T03:47:09.0192496Z github.com/matrixorigin/matrixone/pkg/lockservice.(*server).handle(0xc003b8e4b0, {0x1222f98, 0xc0002afaa0})
2024-05-27T03:47:09.0192812Z 	/home/runner/work/matrixone/matrixone/pkg/lockservice/rpc.go:375 +0x1e7
2024-05-27T03:47:09.0193218Z github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask.func1()
2024-05-27T03:47:09.0193560Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:277 +0xe8
2024-05-27T03:47:09.0194061Z created by github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask in goroutine 49972
2024-05-27T03:47:09.0194410Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:272 +0x167
2024-05-27T03:47:09.0194415Z 
2024-05-27T03:47:09.0194554Z goroutine 50029 [select, 26 minutes]:
2024-05-27T03:47:09.0194999Z github.com/matrixorigin/matrixone/pkg/lockservice.(*server).handle(0xc003b8f2c0, {0x1222f98, 0xc000192030})
2024-05-27T03:47:09.0195323Z 	/home/runner/work/matrixone/matrixone/pkg/lockservice/rpc.go:375 +0x1e7
2024-05-27T03:47:09.0195723Z github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask.func1()
2024-05-27T03:47:09.0196065Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:277 +0xe8
2024-05-27T03:47:09.0196561Z created by github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask in goroutine 49972
2024-05-27T03:47:09.0196910Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:272 +0x167
2024-05-27T03:47:09.0196987Z 
2024-05-27T03:47:09.0197127Z goroutine 50016 [select, 26 minutes]:
2024-05-27T03:47:09.0197570Z github.com/matrixorigin/matrixone/pkg/lockservice.(*server).handle(0xc003b8f2c0, {0x1222f98, 0xc0005b55c0})
2024-05-27T03:47:09.0197889Z 	/home/runner/work/matrixone/matrixone/pkg/lockservice/rpc.go:375 +0x1e7
2024-05-27T03:47:09.0198285Z github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask.func1()
2024-05-27T03:47:09.0198635Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:277 +0xe8
2024-05-27T03:47:09.0199122Z created by github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask in goroutine 49972
2024-05-27T03:47:09.0199479Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:272 +0x167
2024-05-27T03:47:09.0199484Z 
2024-05-27T03:47:09.0199620Z goroutine 49977 [select, 26 minutes]:
2024-05-27T03:47:09.0200102Z github.com/matrixorigin/matrixone/pkg/common/morpc.(*client).createTask(0xc0000de9a0, {0x1222f98, 0xc0003c4960})
2024-05-27T03:47:09.0200452Z 	/home/runner/work/matrixone/matrixone/pkg/common/morpc/client.go:458 +0x113
2024-05-27T03:47:09.0200838Z github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask.func1()
2024-05-27T03:47:09.0201190Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:277 +0xe8
2024-05-27T03:47:09.0201674Z created by github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask in goroutine 49972
2024-05-27T03:47:09.0202025Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:272 +0x167
2024-05-27T03:47:09.0202031Z 
2024-05-27T03:47:09.0202147Z goroutine 50010 [select]:
2024-05-27T03:47:09.0202631Z github.com/matrixorigin/matrixone/pkg/common/morpc.(*client).gcIdleTask(0xc0000debb0, {0x1222f98, 0xc00010ff80})
2024-05-27T03:47:09.0202967Z 	/home/runner/work/matrixone/matrixone/pkg/common/morpc/client.go:383 +0x292
2024-05-27T03:47:09.0203362Z github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask.func1()
2024-05-27T03:47:09.0203699Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:277 +0xe8
2024-05-27T03:47:09.0204317Z created by github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask in goroutine 49972
2024-05-27T03:47:09.0204678Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:272 +0x167
2024-05-27T03:47:09.0204737Z 
2024-05-27T03:47:09.0204877Z goroutine 50018 [select, 26 minutes]:
2024-05-27T03:47:09.0205337Z github.com/matrixorigin/matrixone/pkg/lockservice.(*server).handle(0xc003b8f2c0, {0x1222f98, 0xc0005b5bf0})
2024-05-27T03:47:09.0205654Z 	/home/runner/work/matrixone/matrixone/pkg/lockservice/rpc.go:375 +0x1e7
2024-05-27T03:47:09.0206056Z github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask.func1()
2024-05-27T03:47:09.0206397Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:277 +0xe8
2024-05-27T03:47:09.0206906Z created by github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask in goroutine 49972
2024-05-27T03:47:09.0207251Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:272 +0x167
2024-05-27T03:47:09.0207255Z 
2024-05-27T03:47:09.0207380Z goroutine 50014 [select]:
2024-05-27T03:47:09.0207933Z github.com/matrixorigin/matrixone/pkg/common/morpc.(*server).closeDisconnectedSession(0xc0000dec60, {0x1222f98, 0xc0005b54a0})
2024-05-27T03:47:09.0208279Z 	/home/runner/work/matrixone/matrixone/pkg/common/morpc/server.go:474 +0x1e5
2024-05-27T03:47:09.0208670Z github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask.func1()
2024-05-27T03:47:09.0209019Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:277 +0xe8
2024-05-27T03:47:09.0209515Z created by github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask in goroutine 49972
2024-05-27T03:47:09.0209864Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:272 +0x167
2024-05-27T03:47:09.0209931Z 
2024-05-27T03:47:09.0210072Z goroutine 50025 [select, 26 minutes]:
2024-05-27T03:47:09.0210518Z github.com/matrixorigin/matrixone/pkg/lockservice.(*server).handle(0xc003b8f2c0, {0x1222f98, 0xc0005b5e90})
2024-05-27T03:47:09.0210847Z 	/home/runner/work/matrixone/matrixone/pkg/lockservice/rpc.go:375 +0x1e7
2024-05-27T03:47:09.0211245Z github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask.func1()
2024-05-27T03:47:09.0211598Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:277 +0xe8
2024-05-27T03:47:09.0212092Z created by github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask in goroutine 49972
2024-05-27T03:47:09.0212447Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:272 +0x167
2024-05-27T03:47:09.0212452Z 
2024-05-27T03:47:09.0212588Z goroutine 50030 [select, 26 minutes]:
2024-05-27T03:47:09.0213038Z github.com/matrixorigin/matrixone/pkg/lockservice.(*server).handle(0xc003b8f2c0, {0x1222f98, 0xc000192090})
2024-05-27T03:47:09.0213351Z 	/home/runner/work/matrixone/matrixone/pkg/lockservice/rpc.go:375 +0x1e7
2024-05-27T03:47:09.0213751Z github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask.func1()
2024-05-27T03:47:09.0214104Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:277 +0xe8
2024-05-27T03:47:09.0214592Z created by github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask in goroutine 49972
2024-05-27T03:47:09.0214944Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:272 +0x167
2024-05-27T03:47:09.0214949Z 
2024-05-27T03:47:09.0215062Z goroutine 49982 [select]:
2024-05-27T03:47:09.0215608Z github.com/matrixorigin/matrixone/pkg/lockservice.(*lockTableKeeper).keepRemoteLock(0xc003b8e780, {0x1222f98, 0xc0003c5e90})
2024-05-27T03:47:09.0215992Z 	/home/runner/work/matrixone/matrixone/pkg/lockservice/lock_table_keeper.go:98 +0x3ae
2024-05-27T03:47:09.0216392Z github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask.func1()
2024-05-27T03:47:09.0216737Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:277 +0xe8
2024-05-27T03:47:09.0217350Z created by github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask in goroutine 49972
2024-05-27T03:47:09.0217704Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:272 +0x167
2024-05-27T03:47:09.0217761Z 
2024-05-27T03:47:09.0217901Z goroutine 50027 [select, 26 minutes]:
2024-05-27T03:47:09.0218343Z github.com/matrixorigin/matrixone/pkg/lockservice.(*server).handle(0xc003b8f2c0, {0x1222f98, 0xc0005b5f50})
2024-05-27T03:47:09.0218666Z 	/home/runner/work/matrixone/matrixone/pkg/lockservice/rpc.go:375 +0x1e7
2024-05-27T03:47:09.0219064Z github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask.func1()
2024-05-27T03:47:09.0219405Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:277 +0xe8
2024-05-27T03:47:09.0219900Z created by github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask in goroutine 49972
2024-05-27T03:47:09.0220245Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:272 +0x167
2024-05-27T03:47:09.0220250Z 
2024-05-27T03:47:09.0220394Z goroutine 49987 [select, 26 minutes]:
2024-05-27T03:47:09.0220834Z github.com/matrixorigin/matrixone/pkg/lockservice.(*server).handle(0xc003b8e4b0, {0x1222f98, 0xc0002af7a0})
2024-05-27T03:47:09.0221157Z 	/home/runner/work/matrixone/matrixone/pkg/lockservice/rpc.go:375 +0x1e7
2024-05-27T03:47:09.0221545Z github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask.func1()
2024-05-27T03:47:09.0221891Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:277 +0xe8
2024-05-27T03:47:09.0222378Z created by github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask in goroutine 49972
2024-05-27T03:47:09.0222793Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:272 +0x167
2024-05-27T03:47:09.0222798Z 
2024-05-27T03:47:09.0222934Z goroutine 49991 [select, 26 minutes]:
2024-05-27T03:47:09.0223371Z github.com/matrixorigin/matrixone/pkg/lockservice.(*server).handle(0xc003b8e4b0, {0x1222f98, 0xc0002af980})
2024-05-27T03:47:09.0223688Z 	/home/runner/work/matrixone/matrixone/pkg/lockservice/rpc.go:375 +0x1e7
2024-05-27T03:47:09.0224075Z github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask.func1()
2024-05-27T03:47:09.0224424Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:277 +0xe8
2024-05-27T03:47:09.0224908Z created by github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask in goroutine 49972
2024-05-27T03:47:09.0225257Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:272 +0x167
2024-05-27T03:47:09.0225262Z 
2024-05-27T03:47:09.0225493Z goroutine 49984 [select, 26 minutes]:
2024-05-27T03:47:09.0225940Z github.com/matrixorigin/matrixone/pkg/lockservice.(*server).handle(0xc003b8e4b0, {0x1222f98, 0xc0003c5f80})
2024-05-27T03:47:09.0226251Z 	/home/runner/work/matrixone/matrixone/pkg/lockservice/rpc.go:375 +0x1e7
2024-05-27T03:47:09.0226644Z github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask.func1()
2024-05-27T03:47:09.0226986Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:277 +0xe8
2024-05-27T03:47:09.0227482Z created by github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask in goroutine 49972
2024-05-27T03:47:09.0227830Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:272 +0x167
2024-05-27T03:47:09.0227835Z 
2024-05-27T03:47:09.0227965Z goroutine 50021 [select, 26 minutes]:
2024-05-27T03:47:09.0228401Z github.com/matrixorigin/matrixone/pkg/lockservice.(*server).handle(0xc003b8f2c0, {0x1222f98, 0xc0005b5d10})
2024-05-27T03:47:09.0228707Z 	/home/runner/work/matrixone/matrixone/pkg/lockservice/rpc.go:375 +0x1e7
2024-05-27T03:47:09.0229101Z github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask.func1()
2024-05-27T03:47:09.0229439Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:277 +0xe8
2024-05-27T03:47:09.0230071Z created by github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask in goroutine 49972
2024-05-27T03:47:09.0230419Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:272 +0x167
2024-05-27T03:47:09.0230488Z 
2024-05-27T03:47:09.0230808Z goroutine 50004 [select]:
2024-05-27T03:47:09.0231440Z github.com/matrixorigin/matrixone/pkg/lockservice.(*waiterEvents).handle(0xc004086bd0, {0x1222f98, 0xc00010fbc0})
2024-05-27T03:47:09.0231864Z 	/home/runner/work/matrixone/matrixone/pkg/lockservice/waiter_events.go:183 +0x2ba
2024-05-27T03:47:09.0232258Z github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask.func1()
2024-05-27T03:47:09.0232687Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:277 +0xe8
2024-05-27T03:47:09.0233188Z created by github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask in goroutine 49972
2024-05-27T03:47:09.0233535Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:272 +0x167
2024-05-27T03:47:09.0233540Z 
2024-05-27T03:47:09.0233682Z goroutine 49995 [select, 26 minutes]:
2024-05-27T03:47:09.0234129Z github.com/matrixorigin/matrixone/pkg/lockservice.(*server).handle(0xc003b8e4b0, {0x1222f98, 0xc0002afb00})
2024-05-27T03:47:09.0234567Z 	/home/runner/work/matrixone/matrixone/pkg/lockservice/rpc.go:375 +0x1e7
2024-05-27T03:47:09.0234999Z github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask.func1()
2024-05-27T03:47:09.0235347Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:277 +0xe8
2024-05-27T03:47:09.0235843Z created by github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask in goroutine 49972
2024-05-27T03:47:09.0236196Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:272 +0x167
2024-05-27T03:47:09.0236293Z 
2024-05-27T03:47:09.0236431Z goroutine 50022 [select, 26 minutes]:
2024-05-27T03:47:09.0236886Z github.com/matrixorigin/matrixone/pkg/lockservice.(*server).handle(0xc003b8f2c0, {0x1222f98, 0xc0005b5d70})
2024-05-27T03:47:09.0237207Z 	/home/runner/work/matrixone/matrixone/pkg/lockservice/rpc.go:375 +0x1e7
2024-05-27T03:47:09.0237605Z github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask.func1()
2024-05-27T03:47:09.0237955Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:277 +0xe8
2024-05-27T03:47:09.0238598Z created by github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask in goroutine 49972
2024-05-27T03:47:09.0239063Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:272 +0x167
2024-05-27T03:47:09.0239068Z 
2024-05-27T03:47:09.0239200Z goroutine 49998 [select, 26 minutes]:
2024-05-27T03:47:09.0239649Z github.com/matrixorigin/matrixone/pkg/lockservice.(*server).handle(0xc003b8e4b0, {0x1222f98, 0xc0002aff20})
2024-05-27T03:47:09.0239966Z 	/home/runner/work/matrixone/matrixone/pkg/lockservice/rpc.go:375 +0x1e7
2024-05-27T03:47:09.0240408Z github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask.func1()
2024-05-27T03:47:09.0240752Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:277 +0xe8
2024-05-27T03:47:09.0241259Z created by github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask in goroutine 49972
2024-05-27T03:47:09.0241643Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:272 +0x167
2024-05-27T03:47:09.0241648Z 
2024-05-27T03:47:09.0241826Z goroutine 50000 [sync.Mutex.Lock, 26 minutes]:
2024-05-27T03:47:09.0241994Z sync.runtime_SemacquireMutex(0x0?, 0x0?, 0x0?)
2024-05-27T03:47:09.0242262Z 	/opt/hostedtoolcache/go/1.21.5/x64/src/runtime/sema.go:77 +0x25
2024-05-27T03:47:09.0242402Z sync.(*Mutex).lockSlow(0xc0035b2e30)
2024-05-27T03:47:09.0242655Z 	/opt/hostedtoolcache/go/1.21.5/x64/src/sync/mutex.go:171 +0x213
2024-05-27T03:47:09.0242786Z sync.(*Mutex).Lock(0xc0035b2e30)
2024-05-27T03:47:09.0243143Z 	/opt/hostedtoolcache/go/1.21.5/x64/src/sync/mutex.go:90 +0x55
2024-05-27T03:47:09.0243277Z sync.(*RWMutex).Lock(0xc0035b2e30)
2024-05-27T03:47:09.0243731Z 	/opt/hostedtoolcache/go/1.21.5/x64/src/sync/rwmutex.go:147 +0x45
2024-05-27T03:47:09.0244397Z github.com/matrixorigin/matrixone/pkg/lockservice.(*mapBasedTxnHolder).getTimeoutRemoveTxn(0xc0035b2e00, 0xc00031e630, {0x0, 0x0?, 0x0}, 0x2540be400)
2024-05-27T03:47:09.0244890Z 	/home/runner/work/matrixone/matrixone/pkg/lockservice/service.go:751 +0x11d
2024-05-27T03:47:09.0245433Z github.com/matrixorigin/matrixone/pkg/lockservice.(*service).unlockTimeoutRemoteTxn(0xc000152780, {0x1222f98, 0xc00010f9e0})
2024-05-27T03:47:09.0245804Z 	/home/runner/work/matrixone/matrixone/pkg/lockservice/service_remote.go:490 +0x28f
2024-05-27T03:47:09.0246202Z github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask.func1()
2024-05-27T03:47:09.0246550Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:277 +0xe8
2024-05-27T03:47:09.0247052Z created by github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask in goroutine 49972
2024-05-27T03:47:09.0247398Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:272 +0x167
2024-05-27T03:47:09.0247403Z 
2024-05-27T03:47:09.0247546Z goroutine 49985 [select, 26 minutes]:
2024-05-27T03:47:09.0247994Z github.com/matrixorigin/matrixone/pkg/lockservice.(*server).handle(0xc003b8e4b0, {0x1222f98, 0xc0001cc000})
2024-05-27T03:47:09.0248309Z 	/home/runner/work/matrixone/matrixone/pkg/lockservice/rpc.go:375 +0x1e7
2024-05-27T03:47:09.0248707Z github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask.func1()
2024-05-27T03:47:09.0249049Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:277 +0xe8
2024-05-27T03:47:09.0249549Z created by github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask in goroutine 49972
2024-05-27T03:47:09.0249986Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:272 +0x167
2024-05-27T03:47:09.0249992Z 
2024-05-27T03:47:09.0250133Z goroutine 50023 [select, 26 minutes]:
2024-05-27T03:47:09.0250581Z github.com/matrixorigin/matrixone/pkg/lockservice.(*server).handle(0xc003b8f2c0, {0x1222f98, 0xc0005b5dd0})
2024-05-27T03:47:09.0250903Z 	/home/runner/work/matrixone/matrixone/pkg/lockservice/rpc.go:375 +0x1e7
2024-05-27T03:47:09.0251291Z github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask.func1()
2024-05-27T03:47:09.0251638Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:277 +0xe8
2024-05-27T03:47:09.0252127Z created by github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask in goroutine 49972
2024-05-27T03:47:09.0252477Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:272 +0x167
2024-05-27T03:47:09.0252484Z 
2024-05-27T03:47:09.0252604Z goroutine 49980 [select]:
2024-05-27T03:47:09.0253155Z github.com/matrixorigin/matrixone/pkg/common/morpc.(*server).closeDisconnectedSession(0xc0000dea50, {0x1222f98, 0xc0003c5da0})
2024-05-27T03:47:09.0253499Z 	/home/runner/work/matrixone/matrixone/pkg/common/morpc/server.go:474 +0x1e5
2024-05-27T03:47:09.0253890Z github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask.func1()
2024-05-27T03:47:09.0254247Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:277 +0xe8
2024-05-27T03:47:09.0254752Z created by github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask in goroutine 49972
2024-05-27T03:47:09.0255107Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:272 +0x167
2024-05-27T03:47:09.0255112Z 
2024-05-27T03:47:09.0255244Z goroutine 49990 [select, 26 minutes]:
2024-05-27T03:47:09.0255694Z github.com/matrixorigin/matrixone/pkg/lockservice.(*server).handle(0xc003b8e4b0, {0x1222f98, 0xc0002af920})
2024-05-27T03:47:09.0256013Z 	/home/runner/work/matrixone/matrixone/pkg/lockservice/rpc.go:375 +0x1e7
2024-05-27T03:47:09.0256416Z github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask.func1()
2024-05-27T03:47:09.0256876Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:277 +0xe8
2024-05-27T03:47:09.0257402Z created by github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask in goroutine 49972
2024-05-27T03:47:09.0257823Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:272 +0x167
2024-05-27T03:47:09.0257828Z 
2024-05-27T03:47:09.0257961Z goroutine 50019 [select, 26 minutes]:
2024-05-27T03:47:09.0258414Z github.com/matrixorigin/matrixone/pkg/lockservice.(*server).handle(0xc003b8f2c0, {0x1222f98, 0xc0005b5c50})
2024-05-27T03:47:09.0258730Z 	/home/runner/work/matrixone/matrixone/pkg/lockservice/rpc.go:375 +0x1e7
2024-05-27T03:47:09.0259130Z github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask.func1()
2024-05-27T03:47:09.0259474Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:277 +0xe8
2024-05-27T03:47:09.0259975Z created by github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask in goroutine 49972
2024-05-27T03:47:09.0260322Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:272 +0x167
2024-05-27T03:47:09.0260326Z 
2024-05-27T03:47:09.0260473Z goroutine 50031 [IO wait, 26 minutes]:
2024-05-27T03:47:09.0260665Z internal/poll.runtime_pollWait(0x7f2548ee0d08, 0x72)
2024-05-27T03:47:09.0260950Z 	/opt/hostedtoolcache/go/1.21.5/x64/src/runtime/netpoll.go:343 +0x85
2024-05-27T03:47:09.0261145Z internal/poll.(*pollDesc).wait(0xc000180e20, 0x1?, 0x0)
2024-05-27T03:47:09.0261490Z 	/opt/hostedtoolcache/go/1.21.5/x64/src/internal/poll/fd_poll_runtime.go:84 +0xb1
2024-05-27T03:47:09.0261630Z internal/poll.(*pollDesc).waitRead(...)
2024-05-27T03:47:09.0261941Z 	/opt/hostedtoolcache/go/1.21.5/x64/src/internal/poll/fd_poll_runtime.go:89
2024-05-27T03:47:09.0262159Z internal/poll.(*FD).Accept(0xc000180e00)
2024-05-27T03:47:09.0262465Z 	/opt/hostedtoolcache/go/1.21.5/x64/src/internal/poll/fd_unix.go:611 +0x425
2024-05-27T03:47:09.0262599Z net.(*netFD).accept(0xc000180e00)
2024-05-27T03:47:09.0262856Z 	/opt/hostedtoolcache/go/1.21.5/x64/src/net/fd_unix.go:172 +0x3e
2024-05-27T03:47:09.0263006Z net.(*UnixListener).accept(0xc0005b4780)
2024-05-27T03:47:09.0263290Z 	/opt/hostedtoolcache/go/1.21.5/x64/src/net/unixsock_posix.go:172 +0x3e
2024-05-27T03:47:09.0263436Z net.(*UnixListener).Accept(0xc0005b4780)
2024-05-27T03:47:09.0263691Z 	/opt/hostedtoolcache/go/1.21.5/x64/src/net/unixsock.go:260 +0x65
2024-05-27T03:47:09.0264018Z github.com/fagongzi/goetty/v2.(*server).doStart.func2({0x1220c70, 0xc0005b4780})
2024-05-27T03:47:09.0264671Z 	/home/runner/go/pkg/mod/github.com/matrixorigin/goetty/v2@v2.0.0-20231122095211-6a25dc9130ca/application.go:309 +0xf0
2024-05-27T03:47:09.0264988Z created by github.com/fagongzi/goetty/v2.(*server).doStart in goroutine 49972
2024-05-27T03:47:09.0265748Z 	/home/runner/go/pkg/mod/github.com/matrixorigin/goetty/v2@v2.0.0-20231122095211-6a25dc9130ca/application.go:365 +0x187
2024-05-27T03:47:09.0265755Z 
2024-05-27T03:47:09.0265878Z goroutine 50013 [select]:
2024-05-27T03:47:09.0266449Z github.com/matrixorigin/matrixone/pkg/lockservice.(*lockTableAllocator).cleanCommitState(0xc0002a8be0, {0x1222f98, 0xc0005b4420})
2024-05-27T03:47:09.0266860Z 	/home/runner/work/matrixone/matrixone/pkg/lockservice/lock_table_allocator.go:506 +0x35c
2024-05-27T03:47:09.0267256Z github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask.func1()
2024-05-27T03:47:09.0267609Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:277 +0xe8
2024-05-27T03:47:09.0268111Z created by github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask in goroutine 49972
2024-05-27T03:47:09.0268460Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:272 +0x167
2024-05-27T03:47:09.0268467Z 
2024-05-27T03:47:09.0268608Z goroutine 50024 [select, 26 minutes]:
2024-05-27T03:47:09.0269051Z github.com/matrixorigin/matrixone/pkg/lockservice.(*server).handle(0xc003b8f2c0, {0x1222f98, 0xc0005b5e30})
2024-05-27T03:47:09.0269518Z 	/home/runner/work/matrixone/matrixone/pkg/lockservice/rpc.go:375 +0x1e7
2024-05-27T03:47:09.0269930Z github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask.func1()
2024-05-27T03:47:09.0270347Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:277 +0xe8
2024-05-27T03:47:09.0270842Z created by github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask in goroutine 49972
2024-05-27T03:47:09.0271199Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:272 +0x167
2024-05-27T03:47:09.0271204Z 
2024-05-27T03:47:09.0271318Z goroutine 49981 [select]:
2024-05-27T03:47:09.0271873Z github.com/matrixorigin/matrixone/pkg/lockservice.(*lockTableKeeper).keepLockTableBind(0xc003b8e780, {0x1222f98, 0xc0003c5e30})
2024-05-27T03:47:09.0272261Z 	/home/runner/work/matrixone/matrixone/pkg/lockservice/lock_table_keeper.go:78 +0x259
2024-05-27T03:47:09.0272652Z github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask.func1()
2024-05-27T03:47:09.0273003Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:277 +0xe8
2024-05-27T03:47:09.0273502Z created by github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask in goroutine 49972
2024-05-27T03:47:09.0273857Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:272 +0x167
2024-05-27T03:47:09.0273862Z 
2024-05-27T03:47:09.0273996Z goroutine 49975 [select, 26 minutes]:
2024-05-27T03:47:09.0274461Z github.com/matrixorigin/matrixone/pkg/lockservice.(*detector).doCheck(0xc004df2480, {0x1222f98, 0xc0003c4750})
2024-05-27T03:47:09.0274800Z 	/home/runner/work/matrixone/matrixone/pkg/lockservice/deadlock.go:123 +0x29d
2024-05-27T03:47:09.0275276Z github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask.func1()
2024-05-27T03:47:09.0275622Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:277 +0xe8
2024-05-27T03:47:09.0276125Z created by github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask in goroutine 49972
2024-05-27T03:47:09.0276467Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:272 +0x167
2024-05-27T03:47:09.0276474Z 
2024-05-27T03:47:09.0276617Z goroutine 50005 [select, 26 minutes]:
2024-05-27T03:47:09.0277167Z github.com/matrixorigin/matrixone/pkg/lockservice.(*service).handleFetchWhoWaitingMe(0xc000152780, {0x1222f98, 0xc00010fc20})
2024-05-27T03:47:09.0277539Z 	/home/runner/work/matrixone/matrixone/pkg/lockservice/service_remote.go:546 +0x13a
2024-05-27T03:47:09.0277939Z github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask.func1()
2024-05-27T03:47:09.0278281Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:277 +0xe8
2024-05-27T03:47:09.0278783Z created by github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask in goroutine 49972
2024-05-27T03:47:09.0279129Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:272 +0x167
2024-05-27T03:47:09.0279133Z 
2024-05-27T03:47:09.0279310Z goroutine 49972 [sync.Mutex.Lock, 26 minutes]:
2024-05-27T03:47:09.0279541Z sync.runtime_SemacquireMutex(0xc0058f97a0?, 0x28?, 0x1024d00?)
2024-05-27T03:47:09.0279808Z 	/opt/hostedtoolcache/go/1.21.5/x64/src/runtime/sema.go:77 +0x25
2024-05-27T03:47:09.0279941Z sync.(*Mutex).lockSlow(0xc0001e08b8)
2024-05-27T03:47:09.0280201Z 	/opt/hostedtoolcache/go/1.21.5/x64/src/sync/mutex.go:171 +0x213
2024-05-27T03:47:09.0280326Z sync.(*Mutex).Lock(0xc0001e08b8)
2024-05-27T03:47:09.0280577Z 	/opt/hostedtoolcache/go/1.21.5/x64/src/sync/mutex.go:90 +0x55
2024-05-27T03:47:09.0280704Z sync.(*RWMutex).Lock(0xc0001e08b8)
2024-05-27T03:47:09.0280976Z 	/opt/hostedtoolcache/go/1.21.5/x64/src/sync/rwmutex.go:147 +0x45
2024-05-27T03:47:09.0281552Z github.com/matrixorigin/matrixone/pkg/lockservice.newActiveTxn({0xc000014f5c, 0x4, 0x4}, {0xc000014f5c, 0x4}, 0xc0003c4480, {0x0, 0x0})
2024-05-27T03:47:09.0281867Z 	/home/runner/work/matrixone/matrixone/pkg/lockservice/txn.go:57 +0x9e
2024-05-27T03:47:09.0282608Z github.com/matrixorigin/matrixone/pkg/lockservice.(*mapBasedTxnHolder).getActiveTxn(0xc0035b2e00, {0xc000014f5c, 0x4, 0x4}, 0x1, {0x0, 0x0})
2024-05-27T03:47:09.0283009Z 	/home/runner/work/matrixone/matrixone/pkg/lockservice/service.go:676 +0x2a5
2024-05-27T03:47:09.0283512Z github.com/matrixorigin/matrixone/pkg/lockservice.(*service).Lock(_, {_, _}, _, {_, _, _}, {0xc000014f5c, 0x4, 0x4}, ...)
2024-05-27T03:47:09.0283849Z 	/home/runner/work/matrixone/matrixone/pkg/lockservice/service.go:149 +0x3e4
2024-05-27T03:47:09.0284496Z github.com/matrixorigin/matrixone/pkg/lockservice.TestLockSuccWithKeepBindTimeout.func1(0xc0002a8be0?, {0xc0000ac090, 0x1, 0xc0002a8be8?})
2024-05-27T03:47:09.0284862Z 	/home/runner/work/matrixone/matrixone/pkg/lockservice/service_test.go:1364 +0x29e
2024-05-27T03:47:09.0285512Z github.com/matrixorigin/matrixone/pkg/lockservice.runLockServiceTestsWithLevel.func1.1({0x1224230?, 0xc0002a8be0}, {0xc00004e5b0, 0x1, 0x0?})
2024-05-27T03:47:09.0285878Z 	/home/runner/work/matrixone/matrixone/pkg/lockservice/service_test.go:3442 +0x1d6
2024-05-27T03:47:09.0286548Z github.com/matrixorigin/matrixone/pkg/lockservice.RunLockServicesForTest(0xff?, {0xc00026de80, 0x1, 0x1?}, 0x3b9aca00?, 0xc0001ffc88, 0x0, {0x0, 0x0, 0x0})
2024-05-27T03:47:09.0286896Z 	/home/runner/work/matrixone/matrixone/pkg/lockservice/test_helper.go:82 +0xc4e
2024-05-27T03:47:09.0287293Z github.com/matrixorigin/matrixone/pkg/lockservice.runLockServiceTestsWithLevel.func1()
2024-05-27T03:47:09.0287663Z 	/home/runner/work/matrixone/matrixone/pkg/lockservice/service_test.go:3433 +0x126
2024-05-27T03:47:09.0287987Z github.com/matrixorigin/matrixone/pkg/common/reuse.RunReuseTests(0xc000276da8)
2024-05-27T03:47:09.0288332Z 	/home/runner/work/matrixone/matrixone/pkg/common/reuse/checker.go:145 +0x5a
2024-05-27T03:47:09.0289190Z github.com/matrixorigin/matrixone/pkg/lockservice.runLockServiceTestsWithLevel({0x121c1c0?, 0xc0065d01a0}, 0xff, {0xc000276e80, 0x1, 0x1}, 0x3b9aca00, 0xc000276e70, 0x0, {0x0, ...})
2024-05-27T03:47:09.0289570Z 	/home/runner/work/matrixone/matrixone/pkg/lockservice/service_test.go:3432 +0x285
2024-05-27T03:47:09.0290023Z github.com/matrixorigin/matrixone/pkg/lockservice.TestLockSuccWithKeepBindTimeout(0xc0065d01a0)
2024-05-27T03:47:09.0290395Z 	/home/runner/work/matrixone/matrixone/pkg/lockservice/service_test.go:1346 +0xc5
2024-05-27T03:47:09.0290540Z testing.tRunner(0xc0065d01a0, 0x10a8850)
2024-05-27T03:47:09.0290833Z 	/opt/hostedtoolcache/go/1.21.5/x64/src/testing/testing.go:1595 +0x262
2024-05-27T03:47:09.0290987Z created by testing.(*T).Run in goroutine 1
2024-05-27T03:47:09.0291276Z 	/opt/hostedtoolcache/go/1.21.5/x64/src/testing/testing.go:1648 +0x846
2024-05-27T03:47:09.0291281Z 
2024-05-27T03:47:09.0291430Z goroutine 49996 [select, 26 minutes]:
2024-05-27T03:47:09.0291888Z github.com/matrixorigin/matrixone/pkg/lockservice.(*server).handle(0xc003b8e4b0, {0x1222f98, 0xc0002afb60})
2024-05-27T03:47:09.0292203Z 	/home/runner/work/matrixone/matrixone/pkg/lockservice/rpc.go:375 +0x1e7
2024-05-27T03:47:09.0292616Z github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask.func1()
2024-05-27T03:47:09.0292970Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:277 +0xe8
2024-05-27T03:47:09.0293469Z created by github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask in goroutine 49972
2024-05-27T03:47:09.0293825Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:272 +0x167
2024-05-27T03:47:09.0293831Z 
2024-05-27T03:47:09.0293963Z goroutine 49988 [select, 26 minutes]:
2024-05-27T03:47:09.0294412Z github.com/matrixorigin/matrixone/pkg/lockservice.(*server).handle(0xc003b8e4b0, {0x1222f98, 0xc0002af830})
2024-05-27T03:47:09.0294724Z 	/home/runner/work/matrixone/matrixone/pkg/lockservice/rpc.go:375 +0x1e7
2024-05-27T03:47:09.0295132Z github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask.func1()
2024-05-27T03:47:09.0295475Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:277 +0xe8
2024-05-27T03:47:09.0296087Z created by github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask in goroutine 49972
2024-05-27T03:47:09.0296497Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:272 +0x167
2024-05-27T03:47:09.0296502Z 
2024-05-27T03:47:09.0296639Z goroutine 49976 [select, 26 minutes]:
2024-05-27T03:47:09.0297108Z github.com/matrixorigin/matrixone/pkg/lockservice.(*detector).doCheck(0xc004df2480, {0x1222f98, 0xc0003c47b0})
2024-05-27T03:47:09.0297450Z 	/home/runner/work/matrixone/matrixone/pkg/lockservice/deadlock.go:123 +0x29d
2024-05-27T03:47:09.0297848Z github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask.func1()
2024-05-27T03:47:09.0298192Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:277 +0xe8
2024-05-27T03:47:09.0298689Z created by github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask in goroutine 49972
2024-05-27T03:47:09.0299037Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:272 +0x167
2024-05-27T03:47:09.0299042Z 
2024-05-27T03:47:09.0299178Z goroutine 49983 [select, 26 minutes]:
2024-05-27T03:47:09.0299618Z github.com/matrixorigin/matrixone/pkg/lockservice.(*server).handle(0xc003b8e4b0, {0x1222f98, 0xc0003c5ef0})
2024-05-27T03:47:09.0299937Z 	/home/runner/work/matrixone/matrixone/pkg/lockservice/rpc.go:375 +0x1e7
2024-05-27T03:47:09.0300329Z github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask.func1()
2024-05-27T03:47:09.0300679Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:277 +0xe8
2024-05-27T03:47:09.0301172Z created by github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask in goroutine 49972
2024-05-27T03:47:09.0301589Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:272 +0x167
2024-05-27T03:47:09.0301594Z 
2024-05-27T03:47:09.0301731Z goroutine 50017 [select, 26 minutes]:
2024-05-27T03:47:09.0302172Z github.com/matrixorigin/matrixone/pkg/lockservice.(*server).handle(0xc003b8f2c0, {0x1222f98, 0xc0005b5b30})
2024-05-27T03:47:09.0302488Z 	/home/runner/work/matrixone/matrixone/pkg/lockservice/rpc.go:375 +0x1e7
2024-05-27T03:47:09.0302878Z github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask.func1()
2024-05-27T03:47:09.0303224Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:277 +0xe8
2024-05-27T03:47:09.0303713Z created by github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask in goroutine 49972
2024-05-27T03:47:09.0304062Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:272 +0x167
2024-05-27T03:47:09.0304067Z 
2024-05-27T03:47:09.0304195Z goroutine 50007 [select, 26 minutes]:
2024-05-27T03:47:09.0304752Z github.com/matrixorigin/matrixone/pkg/lockservice.(*service).handleFetchWhoWaitingMe(0xc000152780, {0x1222f98, 0xc00010fce0})
2024-05-27T03:47:09.0305118Z 	/home/runner/work/matrixone/matrixone/pkg/lockservice/service_remote.go:546 +0x13a
2024-05-27T03:47:09.0305627Z github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask.func1()
2024-05-27T03:47:09.0305974Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:277 +0xe8
2024-05-27T03:47:09.0306484Z created by github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask in goroutine 49972
2024-05-27T03:47:09.0306836Z 	/home/runner/work/matrixone/matrixone/pkg/common/stopper/stopper.go:272 +0x167
2024-05-27T03:47:09.0307098Z FAIL	github.com/matrixorigin/matrixone/pkg/lockservice	1800.097s

@sukki37 sukki37 assigned iamlinjunhong and unassigned aressu1985 May 27, 2024
@sukki37
Copy link
Contributor Author

sukki37 commented May 27, 2024

repro:https://github.com/matrixorigin/matrixone/actions/runs/9253738243/job/25454064143?pr=16416

@LeftHandCold
Copy link
Contributor

https://github.com/matrixorigin/matrixone/actions/runs/9272453620/job/25510275544?pr=16407 repro

@LeftHandCold
Copy link
Contributor

https://github.com/matrixorigin/matrixone/actions/runs/9278911463/job/25530649294?pr=16407 repro

@daviszhen
Copy link
Contributor

daviszhen commented May 29, 2024 •
edited
Loading

https://github.com/matrixorigin/matrixone/actions/runs/9280026194/job/25533597406?pr=16456

image

@sukki37
Copy link
Contributor Author

sukki37 commented May 29, 2024

repro：https://github.com/matrixorigin/matrixone/actions/runs/9284635475/job/25547423392?pr=16432

@xzxiong
Copy link
Contributor

xzxiong commented May 30, 2024

repro: https://github.com/matrixorigin/matrixone/actions/runs/9296231763/job/25584433606?pr=16474

TestLockSuccWithKeepBindTimeout.log

This was referenced May 30, 2024
Merged
Merged
@aressu1985 aressu1985 modified the milestones: 1.2.1, 1.3.0 May 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@aressu1985 aressu1985

Labels
bug/ut kind/bug Something isn't working phase/testing severity/s0 Extreme impact: Cause the application to break down and seriously affect the use
Projects
None yet
Milestone
1.3.0
Development

No branches or pull requests
10 participants
@zhangxu19830126 @xzxiong @ck89119 @LeftHandCold @gouhongshen @aressu1985 @iamlinjunhong @daviszhen @sukki37 @matrix-meow