3.x & 4.x Roadmap

[jordansrowles]

Rowles.LeanCorpus Feature Roadmap

I've discovered that I love to have 3 features in one version, and 50,000 in the next. I need a roadmap to stop feature creep.

• As always, bug/security fixes come between minor releases, so you can expect any number of those.
• Development of the community server / studio will happen alongside releases of the main libraries, and is currently considered pre-release with no release date planned as of yet (ideally I want this roadmap done before).
• This is a guide for me, Jordan. If anyone else wants to do any of these features (even out of order), I'll be happy to review it and squeeze it into the next sensible release.
  • Work will be done in sprints when the opportunities arrive, times are estimated days (usually days off work)
• Research items that have not earned a product commitment stay outside the release cycle until benchmark evidence justifies them.

---

3.x Feature completion

3.x is deliberately completion-focused: finish incomplete feature families, improve observability, and lock down the contracts that 4.x will build on.

3.0.0 - CodecKit & Index Rework ✅ 2026-08-25

This release is deliberately architectural.

• ✅️ Single source of truth for codec versioning
• ✅️ Fix the trailer positioning bug
• ✅️ Strengthen validation and migrations
• ✅️ Complete third-party format registration
• ✅️ Improve caching
• ✅️ Formalise per-index, per-field and per-segment codec parameters
• ✅️ Future-proof CodecKit for optional corpus/field/segment calibration
  • Sampling lifecycle
  • Persisted codec parameters
  • Versioned calibration metadata
  • Deterministic decoding independent of the training implementation
• Isolate Windows filesystem behaviour #59
• MMapDirectory can unmap an IndexInput while it is still being read #60
• Improve Windows filesystem behaviour #61

Note: 3.x does not need trained codecs or automatic codec selection. The goal is to avoid another breaking redesign if those become worthwhile later.

3.1.0 - Facets & Aggregations

Estimated effort: 9 days
Reasoning: 3 approx. data structures plus a lot of facet semantics and integration
Estimated date: Friday 4th September 2026

Basic FacetBucket, FacetsCollector and HistogramBucket exist, but the drill-down and advanced aggregation machinery does not.

• Drill-down facets (DrillDownQuery)
• Drill-sideways facets
• Numeric/date range facets
• Date histograms
• HDR/t-digest percentiles (HDR Histogram / t-digest percentiles #55)
• HyperLogLog cardinality (HyperLogLog++ #54)
• Hierarchical/multi-level facet paths
• Missing-value buckets
• Bucket ordering by count/value
• Min/max/sum/average metric aggregations
• Multi-valued facet semantics
• Top-N/offset handling for large facet sets

Note: Don't drift into an Elasticsearch-style aggregation DSL yet. The collector primitives matter more.

3.2.0 - Geo Shapes

Estimated effort: 9 days

Only GeoPointField (LatLongPoint equivalent), GeoBoundingBoxQuery and GeoDistanceQuery exist today.

• LatLonShape
• Cartesian shapes / XYPoint
• Polygon and line-string support
• Polygon/line queries
• Shape relations: intersects, contains, within, disjoint
• Distance sorting
• Geo distance aggregations/facets
• Dateline handling
• Geometry simplification utilities

3.3.0 - Analysis Enhancements

Estimated effort: 7 days
Reasoning: Lots of individual small filters, but token-graph correctness and the analysis inspector add weight

• Extend LimitTokenCountFilter into an analyser wrapper
• TrimFilter
• StemmedOverride
• ConditionalTokenFilter
• KeywordRepeat
• CodepointCount
• Classic tokeniser (ClassicFilter exists)
• CollationKey analyser
• Query-auto-stop-word analyser
• Token-count analyser wrapper
• PatternCaptureGroupTokenFilter
• FingerprintFilter
• TypeTokenFilter
• CommonGramsQueryFilter
• Unicode normalisation-specific filters
• Scandinavian folding/normalisation
• German normalisation
• Arabic/Persian normalisation
• Analysis inspector/debugger
  • Per-stage terms, positions/position lengths, offsets, token type, keyword flag and payload
  • Token graph/alternate-path visibility where applicable
  • Usable from tests, docs and contributor tooling

Note: Token-graph correctness remains an implementation-quality concern across positions, position lengths, offsets, flattening and phrase/query-time handling.

3.4.0 - Suggestions & Parsers

Estimated effort: 10 days
Reasoning: Parser AST / pipeline is the major work, + 2 parsers and completion work

• ContextSuggester
• FuzzySuggester if it provides a useful surface beyond DidYouMeanSuggester
• StandardQueryParser
• SurroundQueryParser
• Completion payloads/metadata
• Completion weighting and result deduplication
• Prefix/regex completion where missing
• Full parser error positions and structured diagnostics
• Full parser AST exposure
• Formal parser pipeline: syntax -> AST -> processors -> Query
  • Field aliases/permissions
  • Normalisation
  • Synonym/default-field expansion
  • Query rules
  • Validation
  • Minimum-should-match transforms
  • Other semantic rewrites
• Parser extensibility hooks
• Query escaping utilities
• Better recovery/error reporting

3.5.0 - Vector & Range Search

Estimated effort: 7 days
Reasoning: Byte vectors and multi-vector documents are the risky pieces, rest builds on existing

• Byte-vector field + kNN
• Vector similarity threshold
• Float/double range fields
• Multi-vector-per-document support
• Quantisation diagnostics
• Better recall/evaluation tooling

3.6.0 - Alt Indexing & Query Models

Estimated effort: 15 days (probably be extended)
Reasoning: Monster of a release.

• MemoryIndex
• Percolator / MonitorQuery
• Chunked stored fields
• Selectable Direct/BlockTree postings
• NioFS-like directory
• Read-only directory wrapper
• Document classification (Document Classification #45)
• XML parser

Note: This remains slightly miscellaneous, but all of it concerns alternative ways of indexing, querying or storing.

3.7.0 - Telemetry & Diagnostics

Estimated effort: 9 days

• Expand ActivitySource coverage and standardise span names/tags
• Slow-query and slow-indexing detection
• Merge amplification and flush metrics
• Searcher/reader lifecycle metrics
• Cache hit/miss metrics
• HNSW traversal diagnostics
• Collector diagnostics
• Backup/restore telemetry
• Segment/index statistics
• Diagnostic snapshot and structured dump APIs
• Sampling and metric-cardinality controls
• Stable LeanCorpus telemetry conventions
• Structured query execution diagnostics
  • Query/clause hierarchy and segment
  • Actual input/output cardinalities and timings
  • Selected physical strategy: postings, bitmap, iterator, vector path, etc.
  • Required frequencies/positions/doc values
  • Cache and collector participation
  • Competitive-score/skipping behaviour
  • HNSW nodes visited, retries, oversampling and exact-rescore counts
  • Candidate counts and fusion/reranking contribution where applicable

Note: 3.x records what actually happened. 4.x will add what was estimated, planned and rejected, and why.

3.8.0 - Languages & Text Similarity

Estimated effort: 8 days

• Hindi stemmer
• Turkish stemmer
• MinHash
• Word2Vec model consumption/integration
• Language detection
• Additional language normalisers
• Better CJK dictionary tooling
  • Add CLI tooling for interacting with the .jlc files #46

---

4.x Architecture sophistication

4.x is where LeanCorpus starts reasoning about execution as a whole rather than only providing capable individual primitives.

4.0.0 - API & Optimiser Contracts

• Remove accumulated deprecated APIs
• Clean up package boundaries
• Formalise experimental/stable APIs
• Formalise codec extension and index-format versioning contracts
• Revisit vector public APIs before they become permanent
• Revisit query/scorer/collector abstractions around optimiser-facing capabilities
  • Estimated cardinality and execution cost
  • Advance/seek cost characteristics
  • Frequency/position/offset/doc-values requirements
  • Maximum-score / upper-bound support
  • Competitive-score pruning support
  • Monotonicity where relevant
  • Cacheability and deterministic cache identity
  • Segment-local resource/materialisation requirements

Goal: custom extensions should be able to participate in optimisation rather than silently forcing opaque slow paths.

4.1.0 - Query Planning & Execution

• Logical and physical query plans
• Cost model distinct from selectivity/cardinality
• Clause/filter ordering
• Segment pruning
• Collector-aware planning
• Adaptive physical strategy selection
  • Bitmap vs iterator
  • Exact vs HNSW vector execution
  • Filtered HNSW strategy
  • Materialisation vs streaming where useful
• Physical-plan explanation and planner tracing
• Planned-vs-observed cardinality/cost reporting
• Alternative-plan rejection diagnostics

Note: There should be one plan model, not separate competing "execution plan" concepts. Logical intent, chosen physical operators, estimates and observed execution should be different layers/views of the same query execution model.

4.2.0 - Index Lifecycle & Integrity

• Point-in-time snapshots and retention
• Backup verification without restoring
• Offline index and segment integrity checking
• Safe repair tooling
• Disk-space watermarks
• Merge throttling and I/O budgets
• Format migration tooling
• Previous-version compatibility verification
• Crash-injection tests
• Interrupted merge/backup hardening
• Corruption simulation

4.3.0 - Retrieval, Ranking & Relevance Unification

Retrieval

• First-class retrieval pipeline for lexical, vector, filtering, fusion and reranking
• Named/configurable retrieval profiles
• Explicit per-stage candidate budgets, topK and oversampling
• Weighted RRF and common fusion contracts
• Per-child candidate windows
• Deterministic tie-breaking and cache identity
• Unified exact/HNSW/filtered-HNSW/reranking strategy selection
• Bounded expensive ranking stages rather than whole-corpus scoring
• Retrieval diagnostics: strategy, candidates, fusion contribution and reranking impact

Ranking

• Ordered/lexicographic ranking criteria
• Non-compensatory ranking where earlier criteria can categorically dominate later ones
• Ability to combine ordered criteria with existing score-based ranking
• Deterministic tie resolution and ranking-profile cache identity
• Criterion-contribution diagnostics

Examples include exact symbol > prefix > fuzzy > semantic, or exact phrase > proximity > loose term match. This must remain a generic primitive rather than a code-search-specific policy.

Relevance evaluation

• Judgement-driven experiment harness over the existing ranking metrics
• Query sets and qrels/relevance judgements
• Named BM25, vector, hybrid, fusion, reranking and ranking-profile variants
• NDCG@K, Recall@K, MRR, MAP and precision/recall where appropriate
• Latency, allocations, candidate counts and index-size measurements
• Parameter sweeps and ablation comparisons
• Reproducible result snapshots and regression thresholds
• Reuse the Linux kernel code-search E2E as an early relevance + systems reference workload

Note: This release consolidates the retrieval/ranking architecture rather than creating separate query, retriever, fusion and ranking-plan hierarchies for the same problem. ADR016 remains the baseline: do not reintroduce Matryoshka, RaBitQ, PQ or compressed late interaction without a materially different design and new benchmark evidence.

---

Investigations, not release commitments

Keep these outside the numbered roadmap until the evidence is strong enough to promote them:

• Patience/adaptive HNSW termination
• Seeded HNSW entry points, including lexical/structural seeding
• N-gram candidate pruning for wildcard/regex queries
• Sparse neural retrieval over the inverted index
• Late-interaction / multi-vector retrieval beyond the basic 3.x primitive
• Corpus-trained codec implementations and automatic codec selection
• Query-time classifier-driven weighting/fusion
• Specialised code-search ranking/product abstractions

The bar for promotion is a representative spike with measurable benefit over the simpler LeanCorpus baseline in latency, throughput, allocations, memory/index size and relevance/recall where applicable.

---

Out of release cycles

• [epic] Architecture, public boundary, contracts, threat model #48
• Add CLI tooling for interacting with the .jlc files #46
• Lower the barrier of entry for Rowles.Text #53