Skip to content

v0.5.7 — hot-fix · cross-folder lazy SQLite cross-thread error

Choose a tag to compare

View all tags
@danielchen26 danielchen26 released this 06 May 20:56
· 89 commits to master since this release
v0.5.7
669f2dc

skylakegrep 0.5.7 — hot-fix: cross-folder lazy was silently failing in worker thread

0.5.6 introduced the parallel proactive umbrella architecture and
fixed cascade's "SQLite objects can only be used in their creating
thread" error by opening a dedicated SQLite connection inside the
cascade worker thread. The same fix was missed for the two
cross-folder lazy paths: cold + wrong-folder branch
(_run_cross in cli.py) and warm + low-confidence branch (the
post-cascade _LZ.lazy_explore_cross_folder invocation).

Both passed the main-thread conn into a ThreadPoolExecutor
worker. SQLite raised
SQLite objects created in a thread can only be used in that same thread. The object was created in thread id A and this is thread id B
inside lazy_explore_cross_folder's embedding-cache lookup. The
exception was caught by the surrounding try / except and
silently turned into "0 results" — the user saw cross-folder
return empty even when the answer existed in a sibling repo.

Why it didn't show up in 0.5.6 bench

The proactive umbrella's parallel-tier architecture saved us. The
0.5.6 bench (10 Django oracle queries) preserves 4/10 hit-rate
because:

  • The hits come from cascade running inside the cwd-indexed
    Django repo (cascade tier).
  • filename_extend (proactive) handles file-name queries.
  • Cross-folder lazy was additional, not load-bearing for the
    bench.

The user-reported symptom was visible only on the wrong-path
demo:
cd /tmp/skg-empty-cwd && SKYGREP_PROACTIVE_DIRS=/tmp/oss-bench skygrep "django ORM query builder"
which printed lazy cross-folder failed: SQLite objects created in a thread can only be used... to stderr alongside an answer
from filename_extend. The umbrella pattern delivered an answer
in 1 s anyway (proactive tier), but the lazy semantic tier was
a dead path.

What changed in 0.5.7

cli.py:

  1. _run_cwd worker (cold-start branch): now opens
    init_db(db_path) inside the worker, passes that conn to
    lazy_explore_cold_start, closes in finally.
  2. _run_cross worker (cold + wrong-folder branch): same
    pattern — own init_db(db_path), own close.
  3. _warm_cross_in_worker (warm + low-confidence cross-
    folder): same pattern. Defined as a closure that opens its
    own conn, runs lazy_explore_cross_folder, closes; the
    outer ThreadPoolExecutor submits the closure rather than
    the bare lazy_explore_cross_folder with main-thread conn.

Identical pattern to the 0.5.6 cascade-in-worker fix; this
release just applies it consistently across all three
ThreadPool worker call sites.

Verified

  • Wrong-path demo: cd into empty dir,
    SKYGREP_PROACTIVE_DIRS=/tmp/oss-bench, query
    "django ORM query builder" → wall 1 s, cross-folder lazy
    returns 5 cosine-ranked Django files (no SQLite error in
    stderr).
  • Hard bench (10 Django oracle queries, real CLI, fresh DB):
    rg-only 0/10 → auto-trigger 4/10 (preserved from
    0.5.3 / 0.5.6).
  • Pytest 217 / 217.

Compatibility

  • Public API unchanged. cli.py internal closures only.
  • Thread-local SQLite connections are ephemeral; no schema
    changes, no index format change, no behavioural change for
    callers who don't go through cross-folder.
  • 0.5.6-built indexes work without re-indexing.

RED LINE checklist

Per docs/EXISTING-INTELLIGENCE-LAYERS.md:

  1. Layer: Tier 3 proactive umbrella subprocesses
    (lazy_explore_cwd, lazy_explore_cross_folder worker
    wrappers).
  2. Interactions: SQLite via init_db(db_path); same
    embedder instance inside each worker; ThreadPoolExecutor.
  3. Action: PRESERVE — 0.5.7 makes the worker pattern match
    what 0.5.6 already did for cascade. Behaviour returns to
    what was intended in 0.5.6.
  4. Promote/demote history: N/A — this is a defect fix.
  5. New hyperparameters: 0.
  6. Bench through real CLI: ✓ (10 Django oracle, real CLI).
  7. Telemetry footer: unchanged.
  8. Hard timeout: cross-folder 8 s, cascade 30 s — both
    preserved.
  9. Personal-data risk: none — generic placeholder query in
    demo.
  10. Bench receipts: 4/10 preserved; wrong-path wall 1 s.

Pending for 0.6

Unchanged from 0.5.6:

  • σ-noise-band → structural-fallback chain (symbol_channel +
    filename_shortcut + lexical_shortcut RRF instead of cross-
    encoder rerank when σ-gap is in the noise band).
  • True streaming inside Ollama batch embed (split 25 → 5 + 20).
  • Structured per-tier observability for ralph / autopilot.
Assets 4
Loading