Indexing hangs / super-linear CPU on realistic C# files — `InstallScriptTests.cs` (2,314 lines, 87 KiB) takes ~223s to index, full repo self-index stalls past 15 min

[Widthdom]

Summary

Indexing the cdidx repo itself with the published v1.11.0 binary does not complete in a reasonable time. The first time I ran cdidx /home/user/CodeIndex --db /tmp/smoke.db I let it run for over 15 minutes at 100% CPU before killing it — the progress bar never advanced past 56.8% [50/88] and the SQLite WAL stopped growing minutes into the run.

Bisecting narrows the worst contributor to one file: tests/CodeIndex.Tests/InstallScriptTests.cs (2,314 lines, 87 KiB). Indexing just that single file in isolation takes ~223 seconds. The time grows strongly super-linearly with file length:

head -N of InstallScriptTests.cs	time to index
500 lines	4s
1,000 lines	13s
1,500 lines	45s
2,000 lines	123s
2,314 lines	223s

t(N) / N roughly doubles per +500 lines, so this is worse than O(N²) on cdidx's own realistic source content. src/CodeIndex/Indexer/SymbolExtractor.cs (10,692 lines) shows a milder version of the same (59s end-to-end on that file alone, ~10s for first 2,000 lines / ~41s for first 4,000 — classic quadratic shape).

For any user who points cdidx at an even modestly-sized repo containing C# 11 raw-string test fixtures with embedded shell heredocs, indexing effectively hangs. This also means cdidx currently cannot finish indexing its own repository with the published binary — a fairly direct self-improvement-loop concern given CLAUDE.md explicitly says review is done against the locally built binary indexing the project itself.

For what it's worth, the same v1.11.0 binary indexes /usr/lib/python3.12 (577 files / 4,388 chunks / 19,948 symbols / 61,421 refs) in about 19 seconds with zero issues, so the regression is specific to certain C# content rather than a general scaling problem.

Repro

# v1.11.0 from install.sh
CDIDX="$(readlink -f "$HOME/.local/bin/cdidx")"
"$CDIDX" --version     # cdidx v1.11.0

git clone https://github.com/Widthdom/CodeIndex /tmp/ci
# Whole-repo self-index: does not complete in 15 min on a 100% CPU core
rm -f /tmp/smoke.db*
timeout 900 "$CDIDX" /tmp/ci --db /tmp/smoke.db
# (process sits at 56.8% [50/88], WAL stops growing, stays pegged at ~100% CPU)

# Single-file repro — just InstallScriptTests.cs:
mkdir /tmp/one && cp /tmp/ci/tests/CodeIndex.Tests/InstallScriptTests.cs /tmp/one/
rm -f /tmp/one.db*
time "$CDIDX" /tmp/one --db /tmp/one.db
# real ~223s

Bisected root content

The pathological file is tests/CodeIndex.Tests/InstallScriptTests.cs. By running cdidx on contiguous ranges of that file, the slowdown tracks with any run that includes lines 146–182 as a prefix plus any substantial amount of trailing content — lines 146–182 in isolation only take ~6s even with 500 lines of trailing content, but appending that prefix to the full 579–2314 tail turns ~3s into 223s.

The content at lines 146–182 is the first C# 11 raw string literal in the file with a shell heredoc nested inside the raw string:

var (exitCode, stdout, stderr) = RunInstallerSnippet(
    $$"""
    detect_platform() { OS_NAME="linux"; ARCH_NAME="x64"; RID="linux-x64"; }
    ...
    mkdir -p "{{installDir}}"
    cat > "{{Path.Combine(installDir, "cdidx")}}" <<'EOF'
    #!/usr/bin/env bash
    echo "cdidx v0.0.0"
    EOF
    chmod +x "{{Path.Combine(installDir, "cdidx")}}"
    ...
    """,
    ...);

Two sanity checks rule out the obvious explanations:

1. Plain "many raw strings" does not reproduce. I generated a synthetic C# file with 25/50/100/200 copies of a similar $$"""...""" block containing bash + dollar-sign content; indexing stayed flat at 1–2 seconds even at 2,800 lines. So it's not simply the count of raw-string literals.
2. Plain "many heredocs inside raw strings" does not reproduce. A minimal file with the exact <<'EOF' ... EOF heredoc-in-raw-string pattern + 4,000 pad lines also completes in 2 seconds.

What does reproduce is combining lines 146–182 with the rest of the file: the combined workload triggers dramatic super-linear scaling, while either piece alone is fine. So it's a content interaction between the early heredoc-in-raw-string block and later content, not the presence of either pattern by itself. Strongly suggestive of a regex in the symbol / reference extractor that stops anchoring correctly once a <<'EOF' / """-adjacent token appears, then backtracks against the remaining file for every subsequent candidate match.

Diagnostics during the stall

• ps shows the cdidx process at ~95% CPU for the entire stall.
• /proc/$pid/fd/ shows no source file open during the stall — all open fds are the binary itself, pipes, and the SQLite files. The file is already memory-resident; the work is pure CPU.
• The SQLite WAL does not grow during long CPU stretches, i.e. no batches are committing. Progress is not I/O-bound.
• CHANGELOG.md / CLAUDE.md for v1.11.0 do not mention a known indexing-performance regression.

Why this matters

• Self-hosting is broken on the shipping binary. CLAUDE.md specifies that review uses dotnet ./src/CodeIndex/bin/Debug/net8.0/cdidx.dll against the project itself. The release binary in install.sh cannot finish that index in any reasonable time on the current main/codex/final-docs-changelog-readme head. Any AI reviewer following CLOUD_BOOTSTRAP_PROMPT.md will land on this wall the moment they try to dogfood.
• Silent failure shape. There is no error, no log, no warning. The progress bar stalls at a specific percentage and CPU stays hot. A user who kills the process after a few minutes will conclude cdidx hangs on "any large repo," when in reality it's a specific content interaction.
• This effectively prevents dogfooding tests/CodeIndex.Tests/ — which is one of the most important places to dogfood, since the InstallScriptTests.cs file is new in v1.11.0 and grew considerably (it is 87 KiB on codex/final-docs-changelog-readme).

Suspected direction (not verified — no local .NET SDK)

Given the strongly super-linear shape and the content interaction, the most likely location is a regex in src/CodeIndex/Indexer/SymbolExtractor.cs or src/CodeIndex/Indexer/ReferenceExtractor.cs with one of:

• an unanchored .* / .*? / [\\s\\S]* that spans across the whole file body once a particular opener (raw-string """ or heredoc <<'EOF') is seen,
• unbounded alternation with overlapping branches creating catastrophic backtracking on shell-like content ($1, $@, $#, {{...}}, nested quotes), or
• a brace/body-range tracker that, once it loses balance on the nested {{...}} interpolations, re-scans for a closing delimiter against the remainder of the file for every candidate symbol from then on.

A reviewer with a local SDK could confirm by:

1. dotnet run --project src/CodeIndex -- /tmp/one against a test fixture matching InstallScriptTests.cs:146-182 + tail, and
2. taking a CPU profile / perf record to pin down which regex / Regex.Matches / Regex.Match call is dominating.

Scope

• src/CodeIndex/Indexer/SymbolExtractor.cs and/or src/CodeIndex/Indexer/ReferenceExtractor.cs — identify the offending pattern, anchor/bound it, or add an up-front guard that skips symbol/reference extraction inside C# 11 raw string literal spans ("""...""" and $$"""...""") since those are source text, not symbols.
• src/CodeIndex/Indexer/FileIndexer.cs — consider a per-file time budget. Even with the regex bug fixed, a single file taking minutes to extract is never a good outcome and should either emit a WARN and skip, or force-complete with partial extraction, so the whole-repo run can't be held hostage by one file.
• tests/CodeIndex.Tests/SymbolExtractorTests.cs (or a new IndexerPerformanceTests.cs) — add a regression test that indexes a fixture derived from InstallScriptTests.cs's pathological prefix and asserts the extractor finishes within e.g. 5 seconds.
• CHANGELOG.md — Fixed entry (English + Japanese) once the root cause is patched.

Notes for reviewer

• I filed this from a cloud Claude Code session per CLOUD_BOOTSTRAP_PROMPT.md, so I have no local .NET SDK and cannot run the fix myself. The diagnosis is bisect + scaling-table + /proc/$pid/fd/ inspection against the v1.11.0 release binary — nothing requires a rebuild to reproduce.
• All measurements were on one CPU core of a linux-x64 container. The absolute times will differ on beefier machines, but the O(N²)-ish shape does not.

Environment

• cdidx: v1.11.0 (from install.sh — readlink -f "$HOME/.local/bin/cdidx").
• Platform: linux-x64 container, single-core-bound CPU work.
• Filed from a cloud Claude Code session per CLOUD_BOOTSTRAP_PROMPT.md.

[Widthdom]

Started working on this on branch fix/issue-447-cs-raw-string-indexing-perf.

[Widthdom]

着手します。ローカルで fix-447 ブランチを切って調査と実装を始めました。まず CLAUDE.md / SELF_IMPROVEMENT.md に沿ってインデックスを最新化し、最新コミットのローカルビルド cdidx で自己ホスト再現と原因特定を進めます。