kash v2.0.0

What's Changed

Added

• Corpus profile. kash build now measures your documents and writes
  data/domain.profile.json: the diacritic mode, structural numbering patterns,
  title stopwords, stem-vowel folding, and — via one model call — the extraction
  vocabulary, priorities and honorifics. Configuration is layered
  defaults < profile < agent.yaml, so agent.yaml is an override you may
  never need to open.
• kash verify audits the provenance chain. It walks the graph, fetches
  each fact’s chunk from the vector store, and reports how much of the graph
  can be shown to a reader in the passage it came from — with examples of
  what cannot. On a real corpus the answer is a number, not a hope.
• kash profile shows what was measured, the evidence behind each decision,
  and which fields agent.yaml overrides. --dry-run re-derives without
  writing, --no-llm measures only, --refresh regenerates.
• Hybrid retrieval. A pure-Go BM25 index (data/lexical.idx) is fused with
  vector search by Reciprocal Rank Fusion, alongside an exact-reference route
  for queries naming a section or verse number.
• Structure-aware chunking. Documents are split on their heading structure
  with citation breadcrumbs, and chunks carry structural metadata (book,
  heading path, reference numbers, content type).
• Semantic graph retrieval. Entity and relationship descriptions are
  embedded into their own collections, so the knowledge graph can be reached by
  meaning rather than only by exact token overlap. Entity hits seed graph
  traversal.
• Chunk-level provenance, end to end. Every graph fact records the chunk it
  was extracted from, and entities carry the passages their facts came from.
  At query time all three graph routes — traversal, the entity vector store
  and the relationship vector store — return chunk ids, and those passages are
  fetched and fused with vector hits. A semantic match on a generated
  description therefore arrives with the source text behind it, rather than a
  synthesis the reader cannot check.
• Query decomposition splits a question into specific entities and broad
  concepts to seed graph search, with a short-query bypass and an LRU cache so
  most queries cost no extra model call.
• Version stamping. Local builds now report a real version, commit and build
  date; the manifest and profile record the binary that produced them.
• Reasoning effort. Reasoning models can be driven at low, medium or
  high through LLM_REASONING_EFFORT, llm.reasoning_effort in
  ~/.kash/config.yaml, or runtime.llm.reasoning_effort in agent.yaml
  — last wins. It applies to build-time extraction, entity adjudication and
  profiling as well as to serving, and a chat request may override it per call.
  Unset means the parameter is not sent at all, so non-reasoning models are
  unaffected.
• Relationship weight as a corpus-time quality signal. Each canonical triple
  now accumulates a co-occurrence count — the number of distinct chunks the same
  (subject, predicate, object) triple was extracted from. The count is stored in
  the relationship vector-store metadata at build time (weight field on
  RelationshipDoc) and used at retrieval time to apply a log1p(weight) boost
  to graph fact scores in rankFactsByContext, so facts attested in many chapters
  surface above those extracted from a single passage. Corpora built before this
  change degrade gracefully: a missing weight key is treated as w = 1 (uniform
  factor, relative order unchanged). A rebuild is not required but will stamp
  weights on an existing corpus. The boost reads the weight of every candidate
  fact from the graph index, not only of the handful the relationship vector
  store returns, so it reaches the whole result set rather than its first rows.
• Gleaning (iterative extraction). Extraction now performs iterative follow-up
  passes per passage batch to recover facts the model missed due to token limits
  or lost attention on dense passages. Each gleaning pass appends the model's
  previous extraction as an assistant message and sends a continuation prompt
  requesting any explicitly stated facts not yet captured, deduplicating newly
  returned triples against existing extractions. Configured via extraction.glean_rounds
  (default: 1 in DomainConfig, layered via defaults < profile < agent.yaml),
  with early exit whenever a pass yields no new unique triples or an empty array.
  Setting glean_rounds: 0 disables gleaning for single-pass extraction.
  Budget for it: a round that finds something costs a second model call for that
  batch, so a dense corpus approaches double the extraction calls of a single
  pass. The early exit keeps that cost proportional to what is actually
  recovered.

Fixed

• BM25 and entity resolution were dead in every Docker deployment. The
  scaffolded Dockerfile copied three paths while kash serve reads five, so
  lexical.idx and entity_aliases.json never reached the image and hybrid
  retrieval silently degraded to vector-only. It now copies data/ wholesale,
  and the build warns when an existing project's Dockerfile is affected.
• retrieval.top_k never applied. Every caller passed a hardcoded default,
  so the configured value was unreachable on the chat, A2A and dashboard paths.
• kash build deleted every comment in agent.yaml by round-tripping it
  through a map to update the MCP description. Nothing writes agent.yaml now.
• Corrupt PDF text was indexed silently. A PDF whose font subsets carry no
  usable ToUnicode CMap yields glyph indices — valid UTF-8 that decodes as a
  substitution cipher. A text-quality gate now rejects it instead of embedding
  it.
• Truncated extraction responses looked like "no facts". A batch that
  returned unusable output was checkpointed as complete and never revisited.
• A reference pattern without a capture group crashed the build or a query
  with an index-out-of-range panic. Patterns are now validated on compile.
• Build artifacts were walked as source documents, so every build reported
  its own manifest and embedded-store files as "not indexed".
• The lexical index did not record its diacritic mode. A corpus built with
  one mode and served with another tokenised queries differently from its own
  index and returned nothing, with no error. The index now pins its own mode.
• Graph term matching measured length in bytes, discarding short Latin terms
  like "om" while admitting any single Devanagari character.
• Derived honorifics lost their trailing space, which is what separates a
  title from the start of a word. Honorifics are stripped with a plain prefix
  cut, so a mined "śrī " echoed back as "śrī" would have eaten the front of
  every entity beginning with those letters. Values filtered against a supplied
  list now come back in the list's own spelling rather than the model's.
• Release notes never came from the CHANGELOG. The release workflow built a
  dynamic regex as "^## \[" ver "\]", but awk resolves \[ inside a
  string literal to a plain [, leaving the character class ^## [1.0.0] —
  which matches no heading. An empty section falls back to the commit list by
  design, so a release would have shipped commit subjects while appearing to
  honour the CHANGELOG. The heading is now matched as a plain string.
• Reranker responses are bounds-checked; a provider returning an out-of-range
  index previously panicked the request. /health now reports reranker status
  from the real gate rather than a partial one.
• A build could hang forever on one provider call. Every HTTP client was a
  zero-value http.Client and the build passed a context with no deadline, so
  a provider that accepted a request and then stalled — or an intermediary that
  dropped the connection without a RST — blocked the process indefinitely. The
  retry loop was no defence: a call that never returns never returns an error
  either. Provider calls are now bounded by a response-header timeout that
  scales with reasoning effort, and extraction prints a live batch counter so a
  stall no longer looks identical to slow progress.
• Every rerank request carried the whole candidate pool. The pool is sized
  for fusion, which is local and free, so a default top_k shipped 200 chunks
  to a paid API billed per hundred, and a configured top_k: 50 would have
  shipped 1000 and likely been rejected — silently, since a failed rerank falls
  back to cosine order. The reranker now sees the first 100 candidates and the
  rest keep their similarity order behind them.
• Facts with unknown provenance borrowed the first chunk in their batch.
  When no passage in a batch showed evidence for a fact, attribution fell back
  to the first one, which fabricated a citation: the fact printed as
  "[passage 1]" and took the chunk-level ranking boost, both on text that does
  not support it. Unknown provenance now stays unknown, and the fact keeps its
  document citation.
• A fact’s passage citation was never checked. The extractor reports which
  passage it used and that report became the chunk id unconditionally, so a
  misreported index printed a passage citation on text that does not support
  the fact and took the chunk-level ranking boost with it. An evidence check
  existed but ran only when the model declined to answer, which is backwards.
  The claim is now evidence rather than authority: preferred when the passage
  it names mentions the fact, and the batch searched otherwise. Matching folds
  the way the corpus does, so a transliterated name still matches its source.
• Generated entity summaries were indistinguishable from quoted source.
  Descriptions written by a model at build time appeared under the same
  heading style as retrieved passages, so an answer could rest on a summary
  and cite a document that never contains those words. They are now labelled
  as generated orienting context, and the prompt asks for every claim to be
  grounded in a numbered passage or a graph fact.
• Internal chunk IDs reached the prompt as pseudo-citations. A graph fact
  whose supporting chunk was not retrieved printed "chunk: tantra_md_312" —
  a string that looks like a reference to a model instructed to cite inline,
  and that no reader can look up. Such a fact now cites its document alone.
• Entity and relationship search failed outright on a small corpus. Both
  asked for five results unconditionally, and the vector store errors when
  that exceeds the collection size instead of returning what it has. Both
  failures are non-fatal, so the dense graph routes went quiet with only a
  warning. Result counts are now clamped to the collection.
• The citation instruction under-used what retrieval provides. It asked
  only for a filename, while every passage carries a number and a structural
  location (book, chapter, verse). It now asks for both.
• Triple extraction was capped at two facts per chunk. The prompt asked for
  "5-20 triples" per batch of ten passages, which rationed dense passages
  regardless of what they stated — measured at 15.3 triples per batch against
  that ceiling of 20. The budget is now per passage, so a passage contributes
  what it actually contains.

Changed

• kash init scaffolds an override file rather than a questionnaire:
  agent.yaml drops from ~121 lines of domain guesswork to ~78, with the
  derived settings documented as optional overrides.
• Retrieval candidate depth scales with the request instead of a fixed ceiling
  of 40, which previously capped recall for the entire system.
• Result diversity is capped per work rather than per file, so several
  editions of one text no longer compete as separate books — and a question
  genuinely concentrated in one text can be answered from it.

Requires rebuild

Run kash build --rebuild on an existing corpus to pick up the new chunk
boundaries, structural metadata and BM25 index. The build refuses to proceed if
structural rules changed under an already-indexed corpus, rather than leaving
the vector and lexical indexes disagreeing about the same chunk.

Existing Docker projects should replace the per-file COPY lines in their
Dockerfile with COPY data/ /app/data/; kash build warns when this applies.

Several graph changes are softer. A rebuild re-extracts under the per-passage
triple budget and with gleaning, writes entity-level chunk provenance, and
stamps relationship weights. None of it is required — an existing graph stays
correct, just sparser and less traceable, and absent provenance or weights read
as unknown rather than failing.

Installation

Linux (amd64)

curl -L https://github.com/akashicode/kash/releases/download/v2.0.0/kash_linux_amd64.tar.gz | tar -xz
sudo mv kash /usr/local/bin/

Linux (arm64)

curl -L https://github.com/akashicode/kash/releases/download/v2.0.0/kash_linux_arm64.tar.gz | tar -xz
sudo mv kash /usr/local/bin/

macOS (Apple Silicon)

curl -L https://github.com/akashicode/kash/releases/download/v2.0.0/kash_darwin_arm64.tar.gz | tar -xz
sudo mv kash /usr/local/bin/

macOS (Intel)

curl -L https://github.com/akashicode/kash/releases/download/v2.0.0/kash_darwin_amd64.tar.gz | tar -xz
sudo mv kash /usr/local/bin/

Windows (amd64)

Download kash_windows_amd64.zip, extract, and add kash.exe to a directory on your PATH.

---

For full usage instructions, configuration options, and examples, see the README.