TKE checkin: TP CN OOMKilled during TPCC/SSB/TPCH/sysbench load

[aptend]

Summary

MO Checkin Regression On TKE has recently started failing frequently on main/4.0 runs. The visible failure pattern is TP CN OOMKilled during TPCC/SSB/TPCH/sysbench load, followed by client connection loss and lock-table related errors.

Good / Bad Boundary

Known good run:

• Run: https://github.com/matrixorigin/matrixone/actions/runs/26881644689
• Namespace: mo-checkin-regression-24774
• Image: commit-fca451f
• Result: success
• TP CN restarts: 0
• TP CN RSS peak: ~10.1/10.3 GiB
• Go heap peak: ~4.1/4.3 GiB

First observed bad run:

• Run: https://github.com/matrixorigin/matrixone/actions/runs/26886279863
• Namespace: mo-checkin-regression-24803
• Image: commit-0935043
• Result: failure
• TP CN mo-checkin-regression-tp-cn-nnbj8 OOMKilled
• RSS peak: ~13.56 GiB
• Go heap peak: ~10.9 GiB

Later runs showed a similar signature:

• https://github.com/matrixorigin/matrixone/actions/runs/26933239941 (mo-checkin-regression-24798)
• https://github.com/matrixorigin/matrixone/actions/runs/26936440344 (mo-checkin-regression-24792)
• retained namespace mo-checkin-regression-24624

Symptoms

Common test/log symptoms:

• TP CN container exits with OOMKilled / exit 137
• TPCC/SSB/TPCH report Lost connection to MySQL server during query
• sysbench and background tasks report many lock table bind changed errors
• the failure concentrates memory on one TP CN while the peer CN stays much lower

Profile / Metrics Observed

Bad TP CN profile/metrics around failure:

• RSS grows to ~13.5 GiB, close to the 14 GiB container limit
• Go heap grows to ~10-12 GiB range
• malloc:inuse_space / mpool profile has no effective signal in checked windows
• heap profile is dominated by lockservice-related stacks such as holders/waiter queue/btree structures

Infrastructure Notes

No clear evidence that the runner or TP CN resource limit was recently reduced:

• TP CN requests/limits observed as 12Gi / 14Gi
• GOMEMLIMIT=8000MiB
• first good and first bad runs used the same TKE worker nodes for the TP CNs
• retained namespace nodes are SA3.2XLARGE16; node memory pressure was false when checked

CI Note

The CI workflow cleanup behavior is being changed separately so failed checkin namespaces are no longer retained by default: matrixorigin/CI#360

[aptend]

Investigation note with current hypothesis and evidence.

Current conclusion

The evidence points to a generic LOAD planning regression introduced by #24803, not parquet reader memory itself.

The affected checkin workloads load CSV/TBL files, not parquet. However, #24803 changed LOAD external-scan stats for all LOAD formats. That stats change makes large CSV/TBL LOAD statements get classified as ExecTypeAP_ONECN instead of ExecTypeAP_MULTICN. As a result, byte-offset parallel LOAD work that previously spread across 2 TP CNs is concentrated on one CN. That CN then accumulates much more lockservice Go heap state and hits the 14 GiB TKE container limit.

Key evidence: LOAD became one-CN

Before #24803, large LOAD statements were compiled with 2 CNs:

• mo-checkin-regression-24774, ssb_10/lineorder.tbl:
  • compileExternScanParallelReadWrite, len of cnList is 2
  • scopes were built for both TP CN pipeline addresses

After #24803, the first bad run compiled large CSV/TBL LOAD statements with only 1 CN:

• mo-checkin-regression-24803, bad CN nnbj8:
  • tpcc_50/customer.csv: len of cnList is 1, mcpu is 6
  • tpcc_50/order-line.csv: len of cnList is 1, mcpu is 7
  • tpcc_50/stock.csv: len of cnList is 1, mcpu is 7
  • ssb_10/lineorder.tbl: len of cnList is 1, mcpu is 7

The retained later failure mo-checkin-regression-24624 shows the same pattern for TPCH:

• tpch_10/customer.tbl: len of cnList is 1
• tpch_10/lineitem.tbl: len of cnList is 1
• tpch_10/orders.tbl: len of cnList is 1
• tpch_10/partsupp.tbl: len of cnList is 1

The other CN was not dead: it had proxy connections, logtail subscriptions, task heartbeats, and show backend servers later saw both CNs as Working. The issue is the compile-time execution type/CN list path, not Kubernetes pod absence.

Code path

#24803 changed LOAD external scan stats from &plan.Stats{} to makeLoadExternalStats(...) in both load build paths:

• pkg/sql/plan/bind_load.go
• pkg/sql/plan/build_load.go

Current makeLoadExternalStats sets:

• Cost = input file bytes
• Rowsize = 1
• BlockNum = 1
• TableCnt = 1
• Outcnt = 1

That TableCnt = 1 prevents ReCalcNodeStats from calling getExternalStats() for Node_EXTERNAL_SCAN. The old path estimated real external-file stats by reading/listing the file and computing approximately:

• Cost = file_size / first_line_size
• BlockNum = Cost / DefaultBlockMaxRows

For the failing CI files, this changes execution-type classification:

file	old estimated type	new type after #24803 stats
tpcc_50/customer.csv	AP_MULTICN	AP_ONECN
tpcc_50/order-line.csv	AP_MULTICN	AP_ONECN
tpcc_50/stock.csv	AP_MULTICN	AP_ONECN
ssb_10/lineorder.tbl	AP_MULTICN	AP_ONECN
tpch_10/lineitem.tbl	AP_MULTICN	AP_ONECN

Relevant code:

• pkg/sql/plan/stats.go:GetExecType
  • AP_MULTICN requires BlockNum > 512 && Cost > 160000
  • with BlockNum=1, large LOAD falls to AP_ONECN because Cost > 240000
• pkg/sql/compile/compile.go:compileQuery
  • ExecTypeTP and ExecTypeAP_ONECN set c.cnList = current CN only
  • only ExecTypeAP_MULTICN calls getCNList() and spreads work across CNs
• pkg/sql/compile/compile.go:compileExternScanParallelReadWrite
  • splits byte ranges across c.cnList; if len(cnList)=1, all splits stay on one CN

Memory profile interpretation

The extra memory is Go heap, not malloc/mpool/off-heap:

• malloc:inuse_space had 0 points for good, bad, and retained failure windows
• bad CN memory:inuse_space reached ~11 GiB while the peer CN stayed around ~1.6-1.9 GiB

Bad CN heap profiles are dominated by lockservice structures:

• lockservice.(*holders).add
• lockservice.newWaiterQueue
• google/btree.(*items).insertAt
• lockservice.newHolders
• stack path includes sql/colexec/lockop.callNonBlocking

Prometheus metrics also line up:

• good run: TP CN heap ~4.3 GiB, lock hold average sub-second on one CN and ~0.42s on the other
• first bad run: bad CN heap ~10.6 GiB, lock hold observations ~11.1M/5min, average hold ~35s
• later bad run: heap ~12.1 GiB, lock hold observations ~13.36M/5min, average hold ~30.5s

Suggested fix direction

Do not use the LOAD byte-size hint as general optimizer stats. The fix should preserve enough byte-size information for LOAD parallel sizing, while keeping execution-type classification based on real external-file cardinality/block estimates, or otherwise mark large LOAD as multi-CN.

Possible directions:

• Let non-parquet CSV/TBL LOAD keep the old &plan.Stats{} path so getExternalStats() computes real stats.
• For parquet LOAD, avoid BlockNum=1/Outcnt=1 fake stats causing one-CN planning; either estimate block/cardinality safely from file/row-group metadata or keep byte-size hint separate from optimizer stats.
• Add a regression test that a large LOAD external scan is classified as ExecTypeAP_MULTICN, not ExecTypeAP_ONECN.