[Bug]: [tke regression] tpcc 100-1000 test oom.

[Ariznawlll]

Is there an existing issue for the same bug?

• I have checked the existing issues.

Branch Name

mian

Commit ID

e4c504a

Other Environment Information

- Hardware parameters:
- OS type:
- Others:

Actual Behavior

job url: https://github.com/matrixorigin/mo-nightly-regression/actions/runs/9211125528/job/25347041190

log：https://grafana.ci.matrixorigin.cn/explore?panes=%7B%22JM6%22:%7B%22datasource%22:%22loki%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bnamespace%3D%5C%22mo-nightly-regression-20240523%5C%22%7D%20%7C%3D%20%60%60%22,%22queryType%22:%22range%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22loki%22%7D,%22editorMode%22:%22builder%22%7D%5D,%22range%22:%7B%22from%22:%22now-12h%22,%22to%22:%22now%22%7D%7D%7D&schemaVersion=1&orgId=1

profile今日复现再抓

Expected Behavior

No response

Steps to Reproduce

trigger tke test

Additional information

No response

[Ariznawlll]

job url：https://github.com/matrixorigin/mo-nightly-regression/actions/runs/9211125528/job/25364260136

已复现，但是profile太大了此处放不下，麻烦联系我，私发

[jensenojs]

看着ReadBloomFilter占用的内存很多, 请徐鹏大哥帮忙找人看一下吧, @XuPeng-SH

alloc

in use

[jensenojs]

由徐鹏哥确认bf的内存占用是很小的, 从heap上看不出什么特别的东西. 需要二分来找这个bug, 先暂时assign在我头上, 等找到引入的commit再交给对应的同事

[XuPeng-SH]

profile里bloom filter 数据是2g, 也就是是metadata cache的大小，预期是这样，每天的daily也一直是2G，没有变过, cn 的内存limit 是55g, 不是这个原因

[XuPeng-SH]

https://grafana.ci.matrixorigin.cn/explore?panes=%7B%22JPW%22:%7B%22datasource%22:%22loki%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bnamespace%3D%5C%22mo-nightly-regression-20240523%5C%22,%20pod%3D%5C%22nightly-regression-dis-tp-cn-sjxch%5C%22%7D%20%7C~%20%60gc%20.%2Aglobal%60%22,%22queryType%22:%22range%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22loki%22%7D,%22editorMode%22:%22builder%22%7D%5D,%22range%22:%7B%22from%22:%221716533233000%22,%22to%22:%221716534054000%22%7D%7D%7D&schemaVersion=1&orgId=1

[XuPeng-SH]

https://grafana.ci.matrixorigin.cn/explore?panes=%7B%22JPW%22:%7B%22datasource%22:%22loki%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bnamespace%3D%5C%22mo-nightly-regression-20240523%5C%22,%20pod%3D%5C%22nightly-regression-dis-tp-cn-sjxch%5C%22%7D%20%7C~%20%60gc%20.%2Aglobal%60%22,%22queryType%22:%22range%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22loki%22%7D,%22editorMode%22:%22builder%22%7D%5D,%22range%22:%7B%22from%22:%221716533233000%22,%22to%22:%221716534054000%22%7D%7D%7D&schemaVersion=1&orgId=1

heap 在oom前的峰值在20g+, bloom filter 所在的metadata cache大小是2g. cn的内存是55g, 看起来和内存的gc有关系，需要二分查找：

[XuPeng-SH]

二分到5月21号的 #16292 跑tpcc也是oom的，更早之前就oom了

[jensenojs]

may dup with :

• [Bug]: [date 6.6]tke regression: tpcc 500ware 1000threads test cn oom #16741

[jensenojs]

not working on it

[jensenojs]

#16741 之后这个问题应该已经解决了, 辛苦动哥有时间检验一下

@aressu1985

[Ariznawlll]

最近没再出现了，先关闭