pkg/sql/sqlitelogictest/tests/local-prepared/local-prepared_test: TestSqlLiteLogic_testindexdelete1slt_good_0_test failed

[cockroach-teamcity]

pkg/sql/sqlitelogictest/tests/local-prepared/local-prepared_test.TestSqlLiteLogic_testindexdelete1slt_good_0_test failed with artifacts on master @ 48a23bf5d4558c51b9597b04a5ff78d51bb28268:

[21:34:24] --- progress: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3908/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/sqlitelogictest/tests/local-prepared/local-prepared_test_/local-prepared_test.runfiles/com_github_cockroachdb_cockroach/external/com_github_cockroachdb_sqllogictest/test/index/delete/1/slt_good_0.test: 10225 statements
[21:34:26] --- progress: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3908/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/sqlitelogictest/tests/local-prepared/local-prepared_test_/local-prepared_test.runfiles/com_github_cockroachdb_cockroach/external/com_github_cockroachdb_sqllogictest/test/index/delete/1/slt_good_0.test: 10262 statements
[21:34:29] --- progress: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3908/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/sqlitelogictest/tests/local-prepared/local-prepared_test_/local-prepared_test.runfiles/com_github_cockroachdb_cockroach/external/com_github_cockroachdb_sqllogictest/test/index/delete/1/slt_good_0.test: 10265 statements
[21:34:31] --- progress: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3908/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/sqlitelogictest/tests/local-prepared/local-prepared_test_/local-prepared_test.runfiles/com_github_cockroachdb_cockroach/external/com_github_cockroachdb_sqllogictest/test/index/delete/1/slt_good_0.test: 10268 statements
[21:34:33] --- progress: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3908/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/sqlitelogictest/tests/local-prepared/local-prepared_test_/local-prepared_test.runfiles/com_github_cockroachdb_cockroach/external/com_github_cockroachdb_sqllogictest/test/index/delete/1/slt_good_0.test: 10272 statements
[21:34:36] --- progress: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3908/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/sqlitelogictest/tests/local-prepared/local-prepared_test_/local-prepared_test.runfiles/com_github_cockroachdb_cockroach/external/com_github_cockroachdb_sqllogictest/test/index/delete/1/slt_good_0.test: 10277 statements
[21:34:38] --- progress: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3908/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/sqlitelogictest/tests/local-prepared/local-prepared_test_/local-prepared_test.runfiles/com_github_cockroachdb_cockroach/external/com_github_cockroachdb_sqllogictest/test/index/delete/1/slt_good_0.test: 10279 statements
[21:34:41] --- progress: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3908/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/sqlitelogictest/tests/local-prepared/local-prepared_test_/local-prepared_test.runfiles/com_github_cockroachdb_cockroach/external/com_github_cockroachdb_sqllogictest/test/index/delete/1/slt_good_0.test: 10284 statements
[21:34:43] --- progress: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3908/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/sqlitelogictest/tests/local-prepared/local-prepared_test_/local-prepared_test.runfiles/com_github_cockroachdb_cockroach/external/com_github_cockroachdb_sqllogictest/test/index/delete/1/slt_good_0.test: 10321 statements
[21:34:45] --- progress: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3908/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/sqlitelogictest/tests/local-prepared/local-prepared_test_/local-prepared_test.runfiles/com_github_cockroachdb_cockroach/external/com_github_cockroachdb_sqllogictest/test/index/delete/1/slt_good_0.test: 10324 statements
[21:34:47] --- progress: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3908/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/sqlitelogictest/tests/local-prepared/local-prepared_test_/local-prepared_test.runfiles/com_github_cockroachdb_cockroach/external/com_github_cockroachdb_sqllogictest/test/index/delete/1/slt_good_0.test: 10326 statements
[21:34:50] --- progress: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3908/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/sqlitelogictest/tests/local-prepared/local-prepared_test_/local-prepared_test.runfiles/com_github_cockroachdb_cockroach/external/com_github_cockroachdb_sqllogictest/test/index/delete/1/slt_good_0.test: 10331 statements
[21:34:53] --- progress: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3908/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/sqlitelogictest/tests/local-prepared/local-prepared_test_/local-prepared_test.runfiles/com_github_cockroachdb_cockroach/external/com_github_cockroachdb_sqllogictest/test/index/delete/1/slt_good_0.test: 10336 statements
[21:34:55] --- progress: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3908/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/sqlitelogictest/tests/local-prepared/local-prepared_test_/local-prepared_test.runfiles/com_github_cockroachdb_cockroach/external/com_github_cockroachdb_sqllogictest/test/index/delete/1/slt_good_0.test: 10338 statements
[21:34:58] --- progress: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3908/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/sqlitelogictest/tests/local-prepared/local-prepared_test_/local-prepared_test.runfiles/com_github_cockroachdb_cockroach/external/com_github_cockroachdb_sqllogictest/test/index/delete/1/slt_good_0.test: 10343 statements
[21:35:00] --- progress: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3908/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/sqlitelogictest/tests/local-prepared/local-prepared_test_/local-prepared_test.runfiles/com_github_cockroachdb_cockroach/external/com_github_cockroachdb_sqllogictest/test/index/delete/1/slt_good_0.test: 10366 statements
[21:35:02] --- progress: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3908/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/sqlitelogictest/tests/local-prepared/local-prepared_test_/local-prepared_test.runfiles/com_github_cockroachdb_cockroach/external/com_github_cockroachdb_sqllogictest/test/index/delete/1/slt_good_0.test: 10368 statements
[21:35:05] --- progress: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3908/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/sqlitelogictest/tests/local-prepared/local-prepared_test_/local-prepared_test.runfiles/com_github_cockroachdb_cockroach/external/com_github_cockroachdb_sqllogictest/test/index/delete/1/slt_good_0.test: 10370 statements
[21:35:08] --- progress: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3908/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/sqlitelogictest/tests/local-prepared/local-prepared_test_/local-prepared_test.runfiles/com_github_cockroachdb_cockroach/external/com_github_cockroachdb_sqllogictest/test/index/delete/1/slt_good_0.test: 10375 statements
[21:35:10] --- progress: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3908/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/sqlitelogictest/tests/local-prepared/local-prepared_test_/local-prepared_test.runfiles/com_github_cockroachdb_cockroach/external/com_github_cockroachdb_sqllogictest/test/index/delete/1/slt_good_0.test: 10380 statements
[21:35:12] --- progress: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3908/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/sqlitelogictest/tests/local-prepared/local-prepared_test_/local-prepared_test.runfiles/com_github_cockroachdb_cockroach/external/com_github_cockroachdb_sqllogictest/test/index/delete/1/slt_good_0.test: 10382 statements
[21:35:15] --- progress: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3908/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/sqlitelogictest/tests/local-prepared/local-prepared_test_/local-prepared_test.runfiles/com_github_cockroachdb_cockroach/external/com_github_cockroachdb_sqllogictest/test/index/delete/1/slt_good_0.test: 10387 statements
[21:35:17] --- progress: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3908/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/sqlitelogictest/tests/local-prepared/local-prepared_test_/local-prepared_test.runfiles/com_github_cockroachdb_cockroach/external/com_github_cockroachdb_sqllogictest/test/index/delete/1/slt_good_0.test: 10394 statements
[21:35:20] --- progress: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3908/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/sqlitelogictest/tests/local-prepared/local-prepared_test_/local-prepared_test.runfiles/com_github_cockroachdb_cockroach/external/com_github_cockroachdb_sqllogictest/test/index/delete/1/slt_good_0.test: 10397 statements
[21:35:22] --- progress: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3908/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/sqlitelogictest/tests/local-prepared/local-prepared_test_/local-prepared_test.runfiles/com_github_cockroachdb_cockroach/external/com_github_cockroachdb_sqllogictest/test/index/delete/1/slt_good_0.test: 10399 statements
[21:35:25] --- progress: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3908/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/sqlitelogictest/tests/local-prepared/local-prepared_test_/local-prepared_test.runfiles/com_github_cockroachdb_cockroach/external/com_github_cockroachdb_sqllogictest/test/index/delete/1/slt_good_0.test: 10404 statements
[21:35:27] --- progress: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3908/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/sqlitelogictest/tests/local-prepared/local-prepared_test_/local-prepared_test.runfiles/com_github_cockroachdb_cockroach/external/com_github_cockroachdb_sqllogictest/test/index/delete/1/slt_good_0.test: 10406 statements
[21:35:29] --- progress: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3908/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/sqlitelogictest/tests/local-prepared/local-prepared_test_/local-prepared_test.runfiles/com_github_cockroachdb_cockroach/external/com_github_cockroachdb_sqllogictest/test/index/delete/1/slt_good_0.test: 10410 statements
[21:35:33] --- progress: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3908/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/sqlitelogictest/tests/local-prepared/local-prepared_test_/local-prepared_test.runfiles/com_github_cockroachdb_cockroach/external/com_github_cockroachdb_sqllogictest/test/index/delete/1/slt_good_0.test: 10415 statements
[21:35:35] --- progress: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3908/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/sqlitelogictest/tests/local-prepared/local-prepared_test_/local-prepared_test.runfiles/com_github_cockroachdb_cockroach/external/com_github_cockroachdb_sqllogictest/test/index/delete/1/slt_good_0.test: 10527 statements
[21:35:37] --- progress: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3908/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/sqlitelogictest/tests/local-prepared/local-prepared_test_/local-prepared_test.runfiles/com_github_cockroachdb_cockroach/external/com_github_cockroachdb_sqllogictest/test/index/delete/1/slt_good_0.test: 10530 statements
[21:35:39] --- progress: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3908/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/sqlitelogictest/tests/local-prepared/local-prepared_test_/local-prepared_test.runfiles/com_github_cockroachdb_cockroach/external/com_github_cockroachdb_sqllogictest/test/index/delete/1/slt_good_0.test: 10532 statements
[21:35:42] --- progress: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3908/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/sqlitelogictest/tests/local-prepared/local-prepared_test_/local-prepared_test.runfiles/com_github_cockroachdb_cockroach/external/com_github_cockroachdb_sqllogictest/test/index/delete/1/slt_good_0.test: 10536 statements
[21:35:44] --- progress: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3908/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/sqlitelogictest/tests/local-prepared/local-prepared_test_/local-prepared_test.runfiles/com_github_cockroachdb_cockroach/external/com_github_cockroachdb_sqllogictest/test/index/delete/1/slt_good_0.test: 10538 statements
[21:35:47] --- progress: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3908/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/sqlitelogictest/tests/local-prepared/local-prepared_test_/local-prepared_test.runfiles/com_github_cockroachdb_cockroach/external/com_github_cockroachdb_sqllogictest/test/index/delete/1/slt_good_0.test: 10542 statements
[21:35:50] --- progress: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3908/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/sqlitelogictest/tests/local-prepared/local-prepared_test_/local-prepared_test.runfiles/com_github_cockroachdb_cockroach/external/com_github_cockroachdb_sqllogictest/test/index/delete/1/slt_good_0.test: 10544 statements
[21:35:53] --- progress: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3908/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/sqlitelogictest/tests/local-prepared/local-prepared_test_/local-prepared_test.runfiles/com_github_cockroachdb_cockroach/external/com_github_cockroachdb_sqllogictest/test/index/delete/1/slt_good_0.test: 10546 statements
[21:35:57] --- progress: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3908/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/sqlitelogictest/tests/local-prepared/local-prepared_test_/local-prepared_test.runfiles/com_github_cockroachdb_cockroach/external/com_github_cockroachdb_sqllogictest/test/index/delete/1/slt_good_0.test: 10551 statements
[21:35:59] --- progress: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3908/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/sqlitelogictest/tests/local-prepared/local-prepared_test_/local-prepared_test.runfiles/com_github_cockroachdb_cockroach/external/com_github_cockroachdb_sqllogictest/test/index/delete/1/slt_good_0.test: 10553 statements
[21:36:01] --- done: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3908/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/sqlitelogictest/tests/local-prepared/local-prepared_test_/local-prepared_test.runfiles/com_github_cockroachdb_cockroach/external/com_github_cockroachdb_sqllogictest/test/index/delete/1/slt_good_0.test with config local-prepared: 10560 tests, 0 failures
    logic.go:4883: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3908/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/sqlitelogictest/tests/local-prepared/local-prepared_test_/local-prepared_test.runfiles/com_github_cockroachdb_cockroach/external/com_github_cockroachdb_sqllogictest/test/index/delete/1/slt_good_0.test:36342: error while processing
    logic.go:4883: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3908/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/sql/sqlitelogictest/tests/local-prepared/local-prepared_test_/local-prepared_test.runfiles/com_github_cockroachdb_cockroach/external/com_github_cockroachdb_sqllogictest/test/index/delete/1/slt_good_0.test:36342: 
        expected success, but found
        (53200) count-leases: root: memory budget exceeded: 10240 bytes requested, 536864325 currently allocated, 536870912 bytes in budget
        resource.go:26: in NewMemoryBudgetExceededError()
    panic.go:694: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/92697d083285a7e03d99bf958ee61c90/logTestSqlLiteLogic_testindexdelete1slt_good_0_test4126900639
--- FAIL: TestSqlLiteLogic_testindexdelete1slt_good_0_test (3714.72s)

Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

Jira issue: CRDB-66476

[michae2]

/investigate

[github-actions]

Investigation: TestSqlLiteLogic_testindexdelete1slt_good_0_test (local-prepared)

Investigated failure: issue body (TeamCity build 21452578, shard 18 of 48)
Failure SHA: 48a23bf5d4558c51b9597b04a5ff78d51bb28268
Confidence: moderate on the mechanism (SQL memory exhaustion), low/moderate on the root cause

What This Test Does

TestSqlLiteLogic_testindexdelete1slt_good_0_test runs the upstream SQLite
logic-test file index/delete/1/slt_good_0.test against a single in-memory
CockroachDB test server, using the local-prepared config (every statement is
prepared, executed, then deallocated). This is a large file — ~10,560
statements — and this run took 3714.7s (~62 minutes), a big outlier versus
the other files in the same shard (which finish in ~1 minute).

Where the Failure Occurs

The failing statement is at file line 36342. It expected success but got an
OutOfMemory (pgcode 53200) error from an internal query:

(53200) count-leases: root: memory budget exceeded:
        10240 bytes requested, 536864325 currently allocated, 536870912 bytes in budget

Decoding the message (<monitor>: memory budget exceeded from
resource.go:26):

• The exhausted monitor is root — the server-wide root SQL memory monitor.
• Its budget is 512 MiB (536870912 = 512 << 20). The logictest harness gives
  the server at least 320 MiB and this run had 512 MiB
  (logic.go:1713-1722).
• At the moment of failure the root monitor was 99.999% full
  (536,864,325 of 536,870,912 bytes; only ~6.5 KiB free), so a routine 10 KiB
  reservation failed.

count-leases is the debug name of the internal query the lease manager runs
while waiting for old descriptor versions to drain
(count.go:113).
It only needed 10 KiB — it is almost certainly the victim that happened to
request memory when the pool was already exhausted, not the cause.

Importantly, the very next subtest (a fresh server) ran fine, so this is
server-wide SQL memory accumulation confined to this one long-running
subtest, not a durable code bug that reproduces on every run.

Analysis

Hypothesis 1 — SQL memory (root monitor, 512 MiB) exhaustion during an
exceptionally large/long logictest file. (moderate confidence on mechanism)

The root SQL monitor filled to its 512 MiB cap during a ~62-minute,
~10,560-statement single-server run and an unrelated internal query
(count-leases) was the first to be denied. Two sub-variants, neither
distinguishable from the runner log alone:

• Gradual accumulation / leak: some per-statement or per-schema-change SQL
  allocation is not released and slowly grows the root monitor over the long
  run. The local-prepared path prepares/executes/closes each statement
  (logic.go:3906-3916),
  so leaked prepared statements are unlikely; more plausible sinks are
  long-lived server-side caches/subsystems (sql-stats/insights, job records,
  descriptor/lease state).
• Single memory-hungry statement: the statement at line 36342 legitimately
  reserved most of the 512 MiB, and count-leases (fired as a side effect,
  e.g. a lease/version wait) tipped it over.

I could not confirm which, because the in-process server logs and any heap
profiles are written to /artifacts/tmp/.../logTestSqlLiteLogic_... inside the
sandbox and are not included in the published TeamCity artifacts (only
test.log/test.xml per shard). The test.log contains no memory warnings
before the failure.

Hypothesis 2 — the recent count-leases change is responsible. (low
confidence — largely ruled out, see below)

Commit af2c51b163a
("lease: use authoritative liveness in version-wait counts", resolving #172036)
is the most recent change to count.go and switched count-leases to the
synchronous/blocking sqlliveness reader
(count.go:107).
That change matters only when system.lease holds many dead sqlliveness
sessions (a large multi-node backlog — see #172036 / #172095). This failure is a
single-node in-memory server, which has essentially no dead-session backlog,
so the blocking reader stays cache-hot and cheap, and count-leases returns a
tiny result set. It is implausible that this query itself consumed ~512 MiB.
This makes the change's appearance in the error message coincidental.

Recommendations:

• Treat as an SQL-memory-accumulation / heavy-outlier-file issue owned by the
  test's team (T-sql-queries), not as a lease/count-leases bug.
• To pinpoint the sink, reproduce locally and capture heap profiles:
  run just this file under local-prepared and watch the root SQL monitor,
  e.g. ./dev test pkg/sql/sqlitelogictest/tests/local-prepared -f=TestSqlLiteLogic_testindexdelete1slt_good_0_test -v
  (expect it to be slow), with -show-logs and periodic runtime/pprof heap
  dumps, or lower --max-sql-memory to force the OOM sooner.
• If this proves to be a rare, non-reproducing accumulation on an outlier file,
  it may be reasonable to skip/split this specific large file for the
  local-prepared config while the underlying accumulation is investigated.

Related Issues and PRs

• #172036 — schema-change
  version-wait hangs when dead sqlliveness sessions exceed the is_alive cache.
  Fixed by af2c51b163a (present at the failure SHA). Same count-leases code,
  different failure mode (a hang, not OOM); requires a large dead-session
  backlog absent here.
• #172095 — orphaned
  dead-session lease cleanup doesn't scale to large backlogs. Also
  system.lease-related; not applicable to a single-node in-memory test.
• No other TestSqlLiteLogic* "memory budget exceeded" failures were found in
  the issue tracker, suggesting this is currently isolated rather than a
  broad regression.

Timeline

• af2c51b163a (2026-07-06) is the last change to pkg/sql/catalog/lease/count.go
  and is an ancestor of the failure SHA — it introduces the count-leases +
  synchronous-liveness-reader wording seen in the error, but (per Hypothesis 2)
  is unlikely to be causal here.
• The failure SHA 48a23bf5d45 is master @ 2026-08-05; the issue was filed
  2026-08-05 and /investigate was requested 2026-08-06. Only this one
  occurrence is recorded in the thread.

Detailed evidence and investigation notes

Log Excerpts

From shard 18 test.log (the failing shard):

[21:36:01] --- done: .../index/delete/1/slt_good_0.test with config local-prepared: 10560 tests, 0 failures
    logic.go:4883:
        .../index/delete/1/slt_good_0.test:36342: error while processing
    logic.go:4883:
        .../index/delete/1/slt_good_0.test:36342:
        expected success, but found
        (53200) count-leases: root: memory budget exceeded: 10240 bytes requested, 536864325 currently allocated, 536870912 bytes in budget
        resource.go:26: in NewMemoryBudgetExceededError()
    panic.go:694: -- test log scope end --
--- FAIL: TestSqlLiteLogic_testindexdelete1slt_good_0_test (3714.72s)
=== RUN   TestSqlLiteLogic_testindexorderby10slt_good_5_test   <-- next subtest, fresh server, PASSES

grep over the entire shard-18 test.log found no memory/leak/slow/OOM
warnings prior to the failure — the log is almost entirely --- progress lines.

Code References

• count.go:112-165
  — the count-leases txn/query (SetDebugName("count-leases"),
  QueryRowEx(ctx, "count-leases", ...)).
• count.go:104-109
  — the WHERE predicate using crdb_internal.sql_liveness_is_alive(session_id, true)
  (the blocking reader introduced by af2c51b163a).
• resource.go:21-33
  — NewMemoryBudgetExceededError message format.
• logic.go:1713-1722
  — logictest SQL memory pool (≥320 MiB; 512 MiB here).

Things Ruled Out

• count-leases / commit af2c51b163a as the cause: requires a large
  dead-session backlog in system.lease (multi-node, crash-heavy). This is a
  single-node in-memory test server; the blocking liveness reader is cache-hot,
  and the query returns a tiny aggregate. Its 10 KiB request is the victim of an
  already-exhausted pool.
• Leaked prepared statements: the local-prepared runner closes prepared
  statements after use, so accumulation via un-deallocated prepared statements is
  unlikely to be the dominant sink.
• A broad/new regression across logictests: no other sqllogictest OOM
  failures found in the tracker; the failure is on a single outlier file.

Other Failure Occurrences

Only one occurrence in this thread (the issue body). Error signature comparison
across the thread is therefore N/A.

Tooling Feedback

• The most useful missing data was the in-process CRDB server log and any heap
  profile for the failing subtest. The runner writes these under
  /artifacts/tmp/.../logTestSqlLiteLogic_... in the sandbox, but the published
  TeamCity artifacts for this Bazel test contain only per-shard test.log and
  test.xml. Publishing the captured test-scope logs (and, for OOM cases, a heap
  profile) would allow attributing the 512 MiB to a specific allocation site.
• Locating the failing shard required downloading test.xml for many of the 48
  shards one URL at a time. The workflow's Bash allowlist forbids seq/loops, so
  a small helper to fan out fetch-url over shard indices (or a build-level
  index mapping test name → shard) would speed this up.

---

Investigation workflow run: https://github.com/cockroachdb/cockroach/actions/runs/31129552801

Was this investigation helpful? Leave a 👍 or 👎 on this comment.

---

Generated by claude-opus-4-8 at high effort. Use /investigate fable-5 max to go deeper.

Workflow run · Was this investigation helpful? Leave a 👍 or 👎 on this comment.