Skip to content

v1.4.0 — Concept Index Aliases

Choose a tag to compare

@barkain barkain released this 25 Mar 23:11
· 27 commits to main since this release
cf577da

Concept Index Aliases for Semantic Search

Closes #7. The concept index now supports LLM-generated aliases — abbreviations, acronyms, and synonyms — so agents find concepts even when their query doesn't match the primary name.

What's new

  • Ingestion-time aliases — the LLM generates 2-3 aliases per concept during ingestion (e.g., "CycloneDX" → ["CDX", "Cyclone DX"]). Stored in manifests and concepts.json.
  • Alias-aware search — search_concepts and the library-researcher agent check aliases alongside concept names. Zero runtime cost.
  • Hit-rate metrics — new concept_search_hits/concept_search_misses fields in benchmark traces, with a post-hoc classifier wired into the scoring CLI.
  • Navigation diagram — README now includes an ASCII diagram showing how agents navigate the library layers.

Before → After

Query Before After
"CDX" ❌ miss → 5-6 calls, ~5k tokens ✅ hit via alias → 2-3 calls, ~1k tokens
"SBOM" ❌ miss on "Software Bill of Materials" ✅ hit via alias
"Maturity Model" ❌ miss on "SCVS BOM Maturity Model" ✅ hit via alias

Upgrade notes

  • Existing books: re-ingest with --force to generate aliases. Books without aliases continue to work (empty aliases list).
  • concepts.json format change: now {"concept": {"chunks": [...], "aliases": [...]}} instead of {"concept": [...]}. The library-researcher agent handles both.
  • Backward compatible: old manifests deserialize cleanly.

Files changed

  • lib/models.py — aliases field on ConceptEntry and CorpusConceptEntry
  • lib/storage.py — alias-aware search_concepts, updated write_concept_index
  • lib/summariser.py — updated concept extraction prompt and parsing
  • preprocessing/books.py, preprocessing/corpus.py — alias passthrough
  • agents/library-researcher.md — updated prompt to check aliases
  • benchmark/ — hit-rate metrics, classifier, report integration
  • README.md — navigation diagram