Skip to content

v1.4.0 - The Code Mode Update

Choose a tag to compare

View all tags
@github-actions github-actions released this 06 Aug 17:24
· 429 commits to main since this release
v1.4.0
0761bb8

The Code Mode Update

v1.4.0 turns ast-sgrep from a blazing-fast CLI into an in-process, programmatic search engine for coding agents.

The headline: a new ast-sgrep-codemode NAPI addon ships inside every platform package — warm, stateful search sessions with zero CLI spawns, and a typed tool catalog (search, semantic, defs, callers, chain, imports, …) that agents compose in a bounded JavaScript sandbox. One asgrep_codemode call replaces several model round trips: seed by intent, fan out defs/callers in parallel with Promise.all, filter and shape in JS, return only the evidence needed.

Also in this release:

  • 13-language structural pattern surface with native C# and Swift grammars (real tree-sitter-c-sharp, no more Java stand-in), plus C/C++/Kotlin/PHP grammars — all pinned to one shared conformance contract
  • Search & ranking honesty: fusion normalization that respects each producer's real scoring contract, coverage-aware ranking, monotonic generations that kill stale cache/IVF identities, and SIMD-accelerated literal search (~60% faster pipeline)
  • LSP symbol navigation that finally resolves case-mismatched and uppercase identifiers
  • Bounded watch freshness, durability/cache correctness, and a large quality + anti-bloat wave backed by measured release gates

Ships as 7 npm packages — ast-sgrep (launcher), pi-ast-sgrep (Pi extension with Code Mode + /asgrep-doctor, /asgrep-status, /asgrep-index, /asgrep-reindex), and 5 @ast-sgrep/<platform> packages for macOS arm64/x64, Linux arm64/x64, and Windows x64 — each carrying the native binary and the codemode NAPI addon, exact-version matched.

v1.4.0 (2026-08-06)

The next release ships seven pull requests plus direct-to-main hardening. Highlights in one line: a new in-process Code Mode (PTC) API, a 13-language pattern/extraction surface with native C# and Swift grammars, search and ranking correctness (fusion normalization, coverage-aware ranking), LSP symbol navigation that finally handles case-mismatched identifiers, bounded watch freshness, durability/cache correctness, and a large quality + anti-bloat wave with measured release gates.

Capability map

PR Theme Files changed
#14 LSP symbol correctness & compatibility 36
#20 P1 store & search correctness 43
#21 Quality & compatibility batch (measured gates) 159
#22 Fusion normalization & ranking correctness 47
#23 C# + 13-language pattern correctness 54
#25 Anti-bloat cleanup & compatibility hardening 62
#26 ast-sgrep-codemode scaffold (Code Mode / PTC) 976*

* #26's file count is dominated by ~867 fuzz corpus fixtures; the feature surface is ~90 source files.

PR #26 — Code Mode (PTC): in-process programmatic search

Delivered capability: a new in-process ast-sgrep-codemode NAPI addon that turns ast-sgrep into a programmatic tool-calling surface for coding agents — warm sessions, typed tool catalog, zero CLI spawn.

Ships a new ast-sgrep-codemode crate and its NAPI addon (ast-sgrep-codemode.node) inside the existing five @ast-sgrep/<platform> npm packages — same install path as the CLI binary, so pi install gets zero-spawn Code Mode out of the box.

  • CodeModeSession: warm, stateful search session over ast-sgrep-core with a sticky Searcher cache, per-call limits (clamped 1–500), and a soft call budget (default 64) that fails closed.
  • Typed, stringly-dispatched tool catalog: search, semantic, chain, defs, callers, imports, index_status, index_repo, filter_hits, select, catalog_search, catalog_describe.
  • In-plan transforms (filter_hits, select) run as pure JSON projections — no shell, no code execution outside the sandbox.
  • Pi extension integration: Code Mode JS sandbox as primary agent execution, warm parallel batching, session-scoped sticky pool, hardened execution paths.

Representative commits: 4873c0e, 47d595c, 5aab31d.

PR #23 — C# correctness and the 13-language pattern surface

Delivered capability: native C# and Swift grammar support plus a shared nine-language conformance contract, delivered through a table-driven 13-language pattern/extraction surface.

  • Native C# grammar: structural patterns and calls now use real tree-sitter-c-sharp instead of a Java stand-in, covering declarations, properties, local functions, constructors, and invocation expressions (difu.5).
  • Complete Swift support: grammar registration, symbol and import extraction, call ownership, structural patterns, source discovery, module resolution, editor activation (difu.2).
  • More grammars: C/C++/Kotlin/PHP grammars and Ruby singleton_method coverage (difu.3/4/6).
  • One shared conformance contract across all nine languages: parse fidelity, symbols, imports, callers, patterns, spans, and false-positive suppression (difu.1), plus a table-driven 13-language pattern/extract surface.
  • Post-review hardening (pushed during this session): literal LIKE/GLOB metacharacter escaping already landed on main (c2j5); single-character hybrid terms stay substantive and embedding switched to full-rank XOF feature hashing (4e9c981); P0 durability/agent/LSP crash paths hardened (fb2cc6b).

PR #22 — Fusion normalization and ranking correctness

Delivered capability: hybrid scores that respect each producer's real scoring contract — no more dilution by unrelated query terms, with coverage-aware, threshold-safe ranking.

  • Lexical fusion normalization: hybrid scores are normalized against the producer's actual rank-zero RRF ceiling instead of being diluted by total query terms (e2hc.14).
  • Single-character queries stay searchable: every non-empty query term is treated as substantive (945bec3).
  • Def/Caller ceilings derive from the terms that actually match each hit's symbol/callee, removing unmatched-term dilution ([u9fj]).
  • Coverage-aware ranking: pre-truncation keeps coverage in the sort key with a keep*4 pool ([8mb8]), rerank writes consistent scores back into hits ([iva9.8]), zero/non-finite scores can no longer fill the limit ([iva9.4]), and invalid file_filter globs error instead of silently skipping the filter ([iva9.2]).
  • Quoted hybrid queries route to a literal pass; structural-index fused scores are bounded at a calibrated fraction of the pattern channel ([noik]).

Representative commits: d7f3ea9, a9860de, b470c6e.

PR #20 — P1 store & search correctness

Delivered capability: monotonic generation counters that kill stale cache/IVF identities, full 256-bit semantic projections, and a bounded max-latency watch pipeline.

  • Monotonic generations: semantic_data_version and searchable-index generations defeat stale semantic cache and IVF identities after delete/re-add, across connections (44a4).
  • All 256 BLAKE3 sign bits consumed in semantic projection instead of tiling the first 32 (e2hc.13).
  • Bounded watch freshness: a max-latency debounce state machine (quiet-gap coalescing + 3× max-latency bound + .asgrep/sidecar self-event filtering) replaces the unbounded-sustained-stream stall (jsfn).
  • Nested-file-transaction depth tracking with poisoned rollback and synchronous=NORMAL restore on end; meta preserved across clears; UTF-8 path handling.

Representative commits: 100424a, 01cdaad, fe0e655.

PR #14 — LSP symbol correctness & compatibility

Delivered capability: reliable definition/reference navigation for uppercase and case-mismatched symbols, with hardened UTF-16 spans and multi-root handling.

  • Case-insensitive symbol navigation: definition/reference lookup routes through case-insensitive indexed resolution, so uppercase and mixed-case symbols resolve reliably (nuli, z47q).
  • Call-chain nodes with source spelling differing from stored symbol case resolve through the real chain expansion path.
  • UTF-16 span fixes: utf16_span_end no longer eats the next character on pure insertion with a zero-length range (c9os).
  • Multi-root folder binding, readiness, dirty-buffer and sync-error hardening (zblv/x46g, ei0i).

PR #21 — Quality & compatibility batch

Delivered capability: measured quality gates, SIMD-accelerated literal search, weighted RRF fusion with learned weights, mmap-backed IVF, and a typed TypeScript Code Mode API — all backed by hard test evidence.

  • Measured quality gates replace vacuous gates: intended-hit and rank contracts, repaired shared-subset rank correlation, ANN quality exercised on the indexed path ([e2hc.19]).
  • Performance: SIMD literal prefiltering + Rayon work stealing with measured work-span profiling ([e2hc.1]); ~60% faster pipeline and ~60% fewer crates LOC.
  • Ranking honesty: immutable signal provenance and within-signal score margins on every result/JSON surface ([e2hc.2]); weighted RRF runtime fusion with learned weights and Fisher-style sensitivity ([e2hc.4]); strict literal → AST → semantic constraint cascade for unprefixed queries ([e2hc.3]).
  • Retrieval: bounded AST-child embeddings with nearest function/file parent mapping ([e2hc.6]); nonfused hierarchical keyword/AST/semantic agent retrieval with stable node refs ([k7l8.4]); pinned caller/import normalization contract ([7uz6]).
  • Freshness & memory: monotonic freshness identity across caches, sidecars, models, bulk/watch indexing ([e2hc.15]); aligned read-only mmap IVF layout with measured cold/fresh/warm open p99 ([e2hc.9]); minified compact output with deduped paths and hard snippet budgets ([k7l8.7]).
  • Security hardening: MCP sandbox/env-trust and poison fail-closed patterns (436d5c3); forbid(unsafe_code) restored via sealed mmap (96e26af); doctor envelope fails closed when unhealthy (eb5577e).
  • Delivery: independent verification of native npm delivery across macOS arm64/x64, Linux arm64/x64, Windows x64 ([ls6.1]); graph retrieval oracle across four languages and four naming styles ([55hl]); case-equivalent retrieval verified against the real senpi monorepo ([oxbj]).

PR #25 — Anti-bloat cleanup & compatibility hardening

Delivered capability: a Zero Tech Debt sweep that deletes dead surfaces, documents honest performance/grammar facts, and hardens compatibility — while preserving every public API.

  • Zero Tech Debt wave: dead surfaces deleted (orphan passes/ tooling, dead re-export shims, ast_grep_pattern_for_query with zero callers), module_resolve split, CLI/search/store surfaces table-driven.
  • Honesty infrastructure: accurate QUERY_GRAMMAR.md, PERF_INVENTORY.md + docs index, benchmark honesty rules, EPIC evidence records.
  • Pi workflow checker moves to python3 YAML (no Ruby): check:pi-contract, check:pi-release, test:pi-release-gate all green.
  • Public APIs preserved during cleanup (8a96bd5).

Also landing on main since v1.3.2 (ships in v1.4.0)

  • fix(store+search): monotonic semantic_data_version defeats cache+IVF collision (44a4)
  • fix(store): symbols_named case-insensitive + functional index (z47q)
  • fix(store): language-aware resolve_module_path (5wkz)
  • fix(embed): probe and cache Ollama/Cloud embedding dim (tmy6); language-aware doc comment markers (pwfm)
  • fix(literal): escape GLOB/LIKE metacharacters in needles (c2j5)
  • Tests: graph query oracle (55hl), imports mixed-case parity (oxbj), uppercase LSP navigation pins (nuli)
  • CI: durable release assets + cross-compile smoke test, idempotent publish + local preflight
Assets 15
Loading