[Bug]: TXN leak causes the CN to crash.

[sukki37]

Is there an existing issue for the same bug?

• I have checked the existing issues.

Environment

- Version or commit-id (e.g. v0.1.0 or 8b23a93):
- Hardware parameters:
- OS type:
- Others:

Actual Behavior

2023-09-30 19:23:40 | 2023/09/30 11:23:40.974660 +0000 FATAL cn-service found leak txn {"uuid": "", "txn-id": "d0285677f3014c73ad051a0b0bcf1750", "create-at": "2023/09/30 10:46:45.758886 +0000", "create-by": "frontend-session-0xc04a5b4e00"}
-- | --

Expected Behavior

No response

Steps to Reproduce

No response

Additional information

No response

[qingxinhome]

Processing

[daviszhen]

问题定位中

[daviszhen]

问题定位中

[xzxiong]

我们云上的日志，
可通过 grafana 查询loki 采集的日志

登入下方链接，可查询到issue 中提到的日志信息（登录密码请咨询 @xzxiong ）


http://47.97.80.230/explore?orgId=1&left=%7B%22datasource%22:%22P3C839627617F91E0%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22P3C839627617F91E0%22%7D,%22editorMode%22:%22builder%22,%22expr%22:%22%7Bnamespace%3D%5C%22freetier-02%5C%22,%20app%3D%5C%22default-cn%5C%22%7D%20%7C%3D%20%60cn-service%20found%20leak%20txn%60%22,%22queryType%22:%22range%22%7D%5D,%22range%22:%7B%22from%22:%221696071600000%22,%22to%22:%221696075199000%22%7D%7D



[xzxiong]

9月21日，freetier-02 的 default-cn 上是有一个 “写监控”的负载。
是由 mo-agent这个组件产生的，它是负责将 Prometheus采集的metric 写入到 MO中存储。

如果你需要在本地拉起，需要两个组件 Prometheus 和 mo-agent



1. Prometheus


   • 下载， 
https://prometheus.io/download/

   • 配置
 prometheus.yml.tgz
   • 启动
 ./prometheus --config.file prometheus.yml --web.enable-lifecycle

2. mo-agent

   • 仓库: https://github.com/matrixone-cloud/observability
   • 分支: git checkout agent_metric
   • 编译: make agent
   • 启动：./bin/mo-agent -c ./etc/mo-agent.yaml
     • 默认使用 dump 账号访问本地的 mo

[daviszhen]

张旭的pr进去之后，再测试

[daviszhen]

张旭的pr进去之后，再测试

[daviszhen]

张旭的pr进去之后，再测试

[daviszhen]

freetier-02升级有问题，dn修复中。

[daviszhen]

freetier-02升级有问题，dn修复中。

[sukki37]

After the upgrade, the issue hasn't reproduced, so we've downgraded it for further observation.

[sukki37]

repro: http://47.97.80.230/explore?panes=%7B%222Qq%22:%7B%22datasource%22:%22P3C839627617F91E0%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bnamespace%3D%5C%227e87d315-be9e-41b6-bef5-b2dd358a53b4%5C%22,%20pod%3D%5C%22normal-cn-dq6wl%5C%22%7D%20%7C%3D%20%60panic%60%22,%22queryType%22:%22range%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22P3C839627617F91E0%22%7D,%22editorMode%22:%22builder%22%7D%5D,%22range%22:%7B%22from%22:%221698283620000%22,%22to%22:%221698283679000%22%7D%7D%7D&schemaVersion=1&orgId=1

[sukki37]

still repro

[nnsgmsone]

pr已经合入，pr修复了一些可能导致的问题。。看是否还会出现

[nnsgmsone]

在观察一天

[nnsgmsone]

在观察一天。

[heni02]

nightly-7244406e 1.0-dev 还有这个问题 @nnsgmsone

cn crash log: cncrash_moc.log
{"level":"INFO","time":"2023/11/10 07:15:50.638991 +0000","caller":"motrace/syncer.go:89","msg":"Wait signal done."}
panic: found leak txn

goroutine 265 [running]:
go.uber.org/zap/zapcore.CheckWriteAction.OnWrite(0x5?, 0x5?, {0x0?, 0x0?, 0xc0b6d9d8a0?})
/go/pkg/mod/go.uber.org/zap@v1.24.0/zapcore/entry.go:198 +0x65
go.uber.org/zap/zapcore.(*CheckedEntry).Write(0xc02cec7ee0, {0xc017c0af00, 0x5, 0x5})
/go/pkg/mod/go.uber.org/zap@v1.24.0/zapcore/entry.go:264 +0x3ec
github.com/matrixorigin/matrixone/pkg/common/log.(*MOLogger).Log(0xc0b6d9d6a0, {0x2f5533e, 0xe}, {{0x0, 0x0}, 0x5, {0x0, 0x0, 0x0}, 0x0, ...}, ...)
/go/src/github.com/matrixorigin/matrixone/pkg/common/log/logger.go:169 +0x44a
github.com/matrixorigin/matrixone/pkg/common/log.(*MOLogger).Fatal(0x2f4b123?, {0x2f5533e?, 0xc14b91b5189e716b?}, {0xc017c0af00?, 0x4d2e8e0?, 0x64?})
/go/src/github.com/matrixorigin/matrixone/pkg/common/log/logger.go:137 +0x4e
github.com/matrixorigin/matrixone/pkg/cnservice.(*service).getTxnClient.func1.1({0xc046fbe450, 0x10, 0xc01bf4d000?}, {0x0?, 0x0?, 0x4d2e8e0?}, {0xc046fb4300, 0xfd}, {0xc045f3a940, 0x39}, ...)
/go/src/github.com/matrixorigin/matrixone/pkg/cnservice/server.go:576 +0x56b
github.com/matrixorigin/matrixone/pkg/txn/client.(*leakChecker).doCheck(0xc0005690e0)
/go/src/github.com/matrixorigin/matrixone/pkg/txn/client/leak_checker.go:114 +0x2b2
github.com/matrixorigin/matrixone/pkg/txn/client.(*leakChecker).check(0xc0005690e0, {0x367e918, 0xc0017e18c0})
/go/src/github.com/matrixorigin/matrixone/pkg/txn/client/leak_checker.go:98 +0x36
github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask.func1()
/go/src/github.com/matrixorigin/matrixone/pkg/common/stopper/stopper.go:259 +0x79
created by github.com/matrixorigin/matrixone/pkg/common/stopper.(*Stopper).doRunCancelableTask
/go/src/github.com/matrixorigin/matrixone/pkg/common/stopper/stopper.go:254 +0xe5

cn crash log：
cncrash_moc.log

[daviszhen]

http://47.97.80.230/explore?orgId=1&panes=%7B%22aG4%22:%7B%22datasource%22:%22loki%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bnamespace%3D%5C%22freetier-01%5C%22%7D%20%7C%3D%20%60goroutine%20160946%60%22,%22queryType%22:%22range%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22loki%22%7D,%22editorMode%22:%22builder%22%7D%5D,%22range%22:%7B%22from%22:%22now-12h%22,%22to%22:%22now%22%7D%7D,%22aLR%22:%7B%22datasource%22:%22loki%22,%22queries%22:%5B%7B%22expr%22:%22%7Bapp%3D%5C%22default-cn%5C%22,apps_kruise_io_cloneset_instance_id%3D%5C%22kjp5z%5C%22,container%3D%5C%22main%5C%22,controller_revision_hash%3D%5C%22default-cn-7997dc9b66%5C%22,filename%3D%5C%22%2Fvar%2Flog%2Fpods%2Ffreetier-01_default-cn-kjp5z_bd212baf-58d8-4bc3-a4a9-271aacd6d27f%2Fmain%2F9.log%5C%22,job%3D%5C%22freetier-01%2Fdefault-cn%5C%22,lifecycle_apps_kruise_io_state%3D%5C%22Normal%5C%22,matrixone_cloud_cluster%3D%5C%22freetier-01%5C%22,matrixone_cloud_component%3D%5C%22cn%5C%22,matrixone_cloud_main_cluster%3D%5C%22freetier-01%5C%22,matrixone_cloud_profile%3D%5C%22cn.standard%5C%22,matrixorigin_io_component%3D%5C%22CNSet%5C%22,matrixorigin_io_instance%3D%5C%22default%5C%22,matrixorigin_io_namespace%3D%5C%22freetier-01%5C%22,namespace%3D%5C%22freetier-01%5C%22,node_name%3D%5C%22cn-hangzhou.10.3.137.132%5C%22,pod%3D%5C%22default-cn-kjp5z%5C%22,pod_template_hash%3D%5C%227997dc9b66%5C%22,stream%3D%5C%22stderr%5C%22%7D%22,%22queryType%22:%22range%22,%22refId%22:%22log-row-context-query-_0.9749565064064918%22,%22maxLines%22:1000,%22direction%22:%22backward%22,%22datasource%22:%7B%22uid%22:%22loki%22,%22type%22:%22loki%22%7D%7D%5D,%22range%22:%7B%22from%22:%221700018176532%22,%22to%22:%221700018177532%22%7D%7D%7D&schemaVersion=1

[daviszhen]

cn 启动失败。报错显示wait TN ready timeout: context deadline exceeded

http://47.97.80.230/explore?orgId=1&panes=%7B%22aG4%22:%7B%22datasource%22:%22loki%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bnamespace%3D%5C%22freetier-01%5C%22%7D%20%7C%3D%20%60panic%60%22,%22queryType%22:%22range%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22loki%22%7D,%22editorMode%22:%22builder%22%7D%5D,%22range%22:%7B%22from%22:%22now-12h%22,%22to%22:%22now%22%7D%7D,%228O6%22:%7B%22datasource%22:%22loki%22,%22queries%22:%5B%7B%22expr%22:%22%7Bapp%3D%5C%22default-cn%5C%22,%20apps_kruise_io_cloneset_instance_id%3D%5C%22k4tx7%5C%22,%20container%3D%5C%22main%5C%22,%20controller_revision_hash%3D%5C%22default-cn-7997dc9b66%5C%22,%20job%3D%5C%22freetier-01%2Fdefault-cn%5C%22,%20lifecycle_apps_kruise_io_state%3D%5C%22Normal%5C%22,%20matrixone_cloud_cluster%3D%5C%22freetier-01%5C%22,%20matrixone_cloud_component%3D%5C%22cn%5C%22,%20matrixone_cloud_main_cluster%3D%5C%22freetier-01%5C%22,%20matrixone_cloud_profile%3D%5C%22cn.standard%5C%22,%20matrixorigin_io_component%3D%5C%22CNSet%5C%22,%20matrixorigin_io_namespace%3D%5C%22freetier-01%5C%22,%20namespace%3D%5C%22freetier-01%5C%22,%20pod%3D%5C%22default-cn-k4tx7%5C%22%7D%22,%22queryType%22:%22range%22,%22refId%22:%22log-row-context-query-_0.4009359423871903%22,%22maxLines%22:1000,%22direction%22:%22backward%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22loki%22%7D,%22editorMode%22:%22builder%22%7D%5D,%22range%22:%7B%22from%22:%221700108402000%22,%22to%22:%221700108404000%22%7D%7D%7D&schemaVersion=1

[daviszhen]