Search is very slow across many indices, even after indexing finished

[marcinzanko]

XERJ field report: Delphi / Object Pascal codebase on Windows (v1.0.0-rc.96)
Date: 2026-10-19. Goal of the trial: index one large Delphi project plus ~19 open-source Delphi repos and use "reference coding" (search how others solved a problem before writing code).

Verdict from the trial: indexing completes (with effort), but search over the result was not usable, so we stopped. Below is what blocked us, ordered by impact, with the evidence we have and a suggested fix for each.

Environment
XERJ v1.0.0-rc.96, Windows 11, 12 cores, RAM not detected by XERJ (see item 4)
Node: xerj --insecure --data-dir

, single node, bound to 127.0.0.1:9200
Project: ~5,000 source files / 167 MB after ignore rules (Delphi: .pas .dpr .dfm .inc, plus some SQL and XML)
Reference repos (all public, shallow clones): JCL, mORMot2, CnVCL, CnWizards, SynEdit, Graphics32, QuickLib, QuickLogger, QuickDAO, DelphiOpenAI, DEC and others
What worked
autoindex --dry-run is fast (3-8 s) and its numbers were accurate.
.xerjignore worked as documented: 15,020 files / 794 MB became 4,981 files / 167 MB.
Resume from the journal is safe: after a failed run, a re-run skipped the 4,888 files already done.
A node started with --insecure listened on 127.0.0.1 only (we did not audit outbound traffic).
Findings

1. Delphi / Object Pascal is not recognised as code (blocker)
   Autoindex of the 5,000-file project reported code_files=18. Almost all other .pas files went in as plain text (ax_format: txt-lines) or as tabular "datasets" with col_1..col_N fields.
   Same on small repos: QuickDAO (13 .pas files) gave code_files=0; CnWizards code_files=6 of 2,068 files; SynEdit code_files=7 and junk_files=158 of 178 files.
   Consequence: no symbols are extracted, so xerj def, xerj search and xerj code have nothing to work with.
   Suggested fix: add a Pascal grammar (.pas .pp .dpr .dpk .inc, and ideally .dfm as a structured text file), or a regex-based fallback that at least extracts procedure/function/class/unit/type declarations.
2. xerj search fails with HTTP 400 instead of falling back (blocker for us)
   On a corpus without code fields, xerj search --prefix

   "" returns unknown field 'defs' in match_phrase clause; unknown field 'name' in term clause ... strict_unknown_fields.
   The error is technically correct but tells the user nothing about the real cause (the corpus has no code index).
   Suggested fix: detect the missing fields and fall back to body/text/title match, or print "corpus has no code index (ax_format: txt-lines); try ...".

3. Default prefix hides existing data
   xerj search "" answers no match in ax-* and suggests running autoindex, even though 190,000 records existed under another prefix (--prefix kxp-...). The user cannot tell "no data" from "wrong prefix".
   Suggested fix: when ax-* is empty, list the prefixes that do exist on the node and their doc counts.
4. RAM not detected on Windows; defaults overload the node
   Startup says: cannot read this machine's total RAM on windows; memory budgets fall back to conservative defaults. Autoindex then picked 12 scan threads, 12 index workers and --bulk-mb 8 without a memory cap.
   Result: _bulk requests timed out at 300 s; retries ran up to 1,500 s per request; the whole run aborted after 8,305 s (2.3 h) with ~190,000 records written. Re-running with --workers 4 --bulk-mb 2 finished in 39 min.
   Suggested fix: read RAM via GlobalMemoryStatusEx; back off automatically (halve bulk size and workers) after the first _bulk timeout instead of retrying the same payload; cap per-request retry time.
5. Node memory growth, one crash, and an unresponsive node while indexing
   Resident memory of the node: 5.3 GB, then 8.2 GB, then 11-13.8 GB during sequential indexing of several corpora. A restart brought it back to ~3-4 GB.
   One crash: Windows Application log, Event ID 1000, faulting module xerj.exe, while indexing a ~3,800-file repo (JCL, 43 MB). The autoindex client then hung for another 43 min before aborting.
   While indexing, _cluster/health and _search often did not answer within 30-90 s.
   Suggested fix: bound the memtable by default even when RAM is unknown; keep search and health responsive under bulk load (separate pools / priorities).
6. Search is very slow across many indices, even after indexing finished
   Each corpus is split into many indices (one per "dataset", e.g. -source-3, -docs). A query through the wildcard (-*) hit the 30 s timeout on a freshly restarted node (5 of 5 queries), and 90 s timeouts on four of six queries across other corpora.
   A single index answered in 100-330 ms, so the cost seems to scale with the number of indices, not documents.
   Suggested fix: reduce indices per corpus (or a single alias with a dataset field), add a per-query timeout that returns partial results, and document the cost of wildcard queries.
7. Indexing throughput on mid-size repos
   mORMot2 (1,374 files, 33 MB): still running after 3 h 13 min at roughly 3 documents/s with 2 workers; we stopped it at ~28,000 records. Our main project (167 MB) took 2.3 h until the abort plus 39 min for the resume.
   Suggested fix: print files/s and an ETA that is not unknown; document expected throughput for lexical mode.
8. Smaller papercuts
   Exit code 3 ("completed with skipped files") is reported as a failure by common task runners, and junk_files has no explanation: we could not tell which files were junked or why (e.g. 158 of 178 files in one repo). A --explain-junk listing would help.
   After an aborted run .autoindex.lock stays in the state dir and blocks the next run until deleted by hand.
   The docs say nothing about source languages the code index supports; a table would save everyone a trial.
   No hub corpus for Delphi / Pascal, which would be an easy way to make the feature useful for this community.
   Reproduction (minimal)
   git clone --depth 1 https://github.com/exilon/QuickDAO and xerj autoindex --prefix ref-quickdao
   Observe code_files=0; query ref-quickdao-docs mapping: .pas files appear as ax_format: txt-lines.
   xerj search --prefix ref-quickdao "how to connect to database" returns the HTTP 400 above.
   What would make us try again
   (a) Pascal recognised as code, (b) a graceful fallback in xerj search, (c) RAM-aware defaults on Windows, (d) wildcard queries answering in seconds. We can re-test a new build on the same corpora if useful.

[xerj-org]

Issue detected. Delphi parser was not implemented correctly