Release contextlake 9.0.0 · sayak-sarkar/contextlake

Added

contextlake kb keys, the command every key-file refusal already named. Seven verbs:
create, list, show, revoke, rotate, check and prune.

kb serve refuses to start on several key-file states, and each refusal told the operator
to run contextlake kb keys create <name>. That command did not exist. A server that
refuses and names a way out the reader cannot take is worse than one that starts wrongly,
because the operator has nothing to do next. This is that command.

The key is shown once. It goes to standard error at creation and never appears again,
because the file stores a SHA-256 digest rather than the key. It is deliberately never sent
through the logger: the console handler always writes to stdout, a --log-file run adds a
5 MB rotating file with three backups that outlives the process, and the redactor rewrites
workspace paths and repo names, so it would scrub a key on neither. A lost key is rotated,
never recovered, and rotate keeps the old key working for --overlap so the holder can
swap without an outage.

Two other paths can carry the key out, and both are narrower than they look. --print-key
writes the bare key to stdout for a pipe and refuses a terminal, where it would land in the
scrollback instead of a secret store. --out FILE writes it at mode 0600, with the mode set
at creation rather than chmod-ed afterwards, and with O_EXCL so an existing path is
refused rather than overwritten.

check reads the key from standard input only. A key on a command line lands in shell
history and shows in ps to every account on the machine. A terminal with nothing piped
in is refused too, rather than waiting for end-of-file behind a blank screen: it does not
prompt, because a typed key lands in the scrollback. It opens no socket and sends no
request, which is what lets it answer when the server is the thing that is down, and it
says so rather than implying it verified anything against a server.

No verb opens the store database. Every one of them runs on a machine that has never
built an index, which is the machine an operator is on when a server has just refused to
start. kb keys list is the first command they run.

Two refusals split by verb rather than collapsed into one rule. A key file carrying group
or other bits, or sitting in a directory anyone can write to, is a policy fault: write
verbs refuse, and list warns and prints the table anyway, because blocking the operator
from seeing what exists is the wrong failure when list is how they diagnose the refusal
they just hit. A file that cannot be read at all is not that: nothing was read, so every
verb fails and names the path.

The scope flags are recorded and enforced by nothing, and every surface says so.
--tools, --repos, --owners, --rate, --burst and --cost-budget are written onto
the key and rendered back by create, list, show and check. No code reads them. A
key created with --tools none --repos nothing-matches/* was presented to a live
kb serve --transport http --keys-only server and tools/list answered with all 23
registered tools, one of which then ran and returned a result.

So the values print with (recorded, not enforced) beside them and three lines saying
what that means, show --json and list --json carry "policy_enforced": false, and an
unset axis reads unset rather than none, which read as a denial for the key a bare
create makes. Telling an operator their key is scoped when it is not is worse than not
offering the flags, because they hand the key out on that reading.

Scope is per tool, not per repository once it does work: a key allowed a tool will read
every indexed repo through it. --rate and --cost-budget are stored as typed and
validated by nobody, because their parser ships with the rate limiter. The LAST USED
column reads never until the usage file it reads from exists.
Per-request identity on the network MCP transports, and the frame that will carry access
control. Groundwork only: nothing is enforced yet and nothing changes for a local run.

build_http_app now resolves a Principal per request and the tool wrapper reads it, so
two callers holding two different credentials are two different identities inside a tool
body rather than one anonymous caller. Until now there was no place to put that fact, which
is why access control, rate limiting and usage accounting could not be built: each would
have had to invent its own notion of who was asking.

The wrapper's whole try/finally/except structure lands here, once, deliberately. Four
planned stories each need to add a line to that twelve-line function, and when they were
specified separately their orderings contradicted each other. Landing the frame first turns
each of them into an insertion at a named anchor instead of a restructure, so whoever lands
second does not have to unpick whoever landed first.

It fails closed. Whether identity is required is a build-time decision made by
build_http_app, never inferred from whether an identity happens to be present. A stdio run
does not read the value at all, and on a network run a missing identity refuses the call
rather than answering it unscoped. Those two states used to be the same value, which meant a
server whose identity plumbing broke would have answered every request as if unauthenticated
access were intended, with every test still passing.

What it does not do. It detects a MISSING identity. It cannot tell a WRONG one, and on
the SSE transport the plausible failure is substitution rather than absence. Closing that
needs a per-connection token compared at the boundary, which is specified and not built.
ask calls its sibling tools directly, so those legs never cross the wrapper and an access
check placed there will not cover them; the anchor comment says so. And the run outcome the
wrapper records has no reader until the usage recorder lands.

stdio, the default, is unchanged: same tool output, no identity lookup, no new file,
config field or dependency for anyone who never serves over the network.
contextlake kb source wizard. It lists every configured source with a reachability
mark, then offers to add another and loops until you answer no (pressing enter is no). The
survey reads the same verify_source path kb doctor and kb source test use, so
"is this source reachable" has one answer across all three. The add step is kb source add
run interactively, so the prompts, the literal-secret refusal and the write target are the
same ones. It needs a terminal: a prompt written to a pipe hangs, so a non-interactive run
is refused with exit 2 and the flag form to use instead.

Changed

kb enrich reports edges to code, not only documents stored. The linking step already
ran: every enrichment document was matched against the repo's symbol names and the matches
were stored as documented_by edges. The count was then discarded, so the run could report
only how many documents came back. A document with no edge to any symbol cannot answer a
question about the code, and it read as a success.

The run now prints, per repo, the terms tried, the documents returned and the edges attached
to code, and closes with a line that puts every targeted repo in one of five buckets:
enriched, nothing returned, returned but unattached, failed, skipped. The five add up to the
number of repos the run planned to touch, on every exit path.

"Returned but unattached" is a state, not a failure. The matcher is whole-word with a
three-character floor, so a ticket that discusses a repo in prose without naming a symbol
correctly attaches to nothing. That run still prints ✓.

A repo whose store or shard write fails is now counted and reported instead of aborting the
whole run. A run where every repo failed that way exits 1.

API change: run_enrich_repo returns an EnrichCounts(terms, documents, edges) triple
instead of the document count alone.
kb doctor's per-source line now separates three answers that used to render as one
⚠. A source that was dialled and did not answer keeps ⚠. A source of a type with no
reachability probe (gitlab, zendesk) now draws ⊘, matching what kb source test
already prints for the same case. A source with enabled = false also draws ⊘ and is not
dialled at all, matching kb connect and kb ingest, which both skip disabled sources.
None of the three changes doctor's exit code, which is unchanged and still deliberate.

What this loses: doctor no longer reports a broken path or an unreachable endpoint on a
disabled source. Nothing reads that source, so the round trip bought nothing, but the line
used to be there. Re-enable the source to have it dialled again.
The bundled --sample demo fleet moved to a new domain, and two of its repos changed
shape. Repo ids and symbol names changed, so a script that names one has to be edited. The
old fleet's domain read as a real production estate rather than as an obvious invention, which
is what demo data has to be. It now models a weather-station monitoring network: stations
report readings, a forecast service runs a model over them, an ingest pipeline normalises raw
readings, an alerts service fans out severe-weather notices, and a console UI shows it.
```
acme/auth-service   ->  acme/station-registry
acme/catalog-api    ->  acme/forecast-api    (rebuilt, not renamed; see below)
acme/payments-api   ->  acme/sensor-ingest   (call edges redirected; see below)
acme/web-ui         ->  acme/console-ui
acme/notifications  ->  acme/alerts
acme/shared-lib     ->  acme/shared-lib      (unchanged)
demo/app            ->  demo/app             (id unchanged; its two symbols changed)
```
acme/forecast-api is a rebuild. The old repo was a four-layer controller / service /
repository / validator chain in C#. It is now a scheduled model run in Go:
RunCycle -> ForecastRunner -> GridSampler -> ModelGrid, with no controller, no repository and
no validator. acme/sensor-ingest keeps its nodes and redirects its call edges so readings
flow one way: SensorGateway -> Ingest -> ReadingProcessor, plus
Backfill -> ReadingProcessor. Nothing downstream calls back into the gateway, and the two
triggers converge on one processor instead of forming another straight line.

What moved and what did not, measured on the fixture. The acme org, the repo count (7),
the node total (44), the edge total (36) and all 11 cross-repo edges are the same, edge for
edge, under the mapping above. Two counts did move: languages went from 4 to 5 (Go added, C#
down from 10 nodes to 5), and node kinds from 7 to 8 (struct added, class 12 to 11,
method 3 to 2).

Every symbol inside those repos changed with the ids, so a golden-query file, an MCP call or
a dashboard bookmark naming an old one returns nothing until it is updated. The new node and
edge lists are in src/contextlake/kb/dashboard/fixtures/sample-dashboard.json. Nothing in
the product changed. This is demo data and the docs that quote it.
[embeddings] base_url now defaults per provider. Check yours before upgrading if you point
it at a local server. The field was declared as the literal http://127.0.0.1:11434, and one
declared literal wins for every provider. So [embeddings] provider = "openai" with no
base_url line sent each batch of indexed code to POST http://127.0.0.1:11434/embeddings,
with the value of OPENAI_API_KEY in the Authorization header. The field is now unset by
default and resolved when it is read: https://api.openai.com/v1 for openai,
http://127.0.0.1:11434 for everything else. [llm] already worked this way.

This changes where your requests go. If you run an OpenAI-compatible server on port 11434
and reach it with provider = "openai" and no base_url line, that traffic stayed on your
machine before the upgrade. After it, embedding requests go to
https://api.openai.com/v1/embeddings and your indexed code leaves the machine. One line keeps
it where it was:
```
[embeddings]
provider = "openai"
base_url = "http://127.0.0.1:11434"
```
Nothing changes for provider = "ollama", "builtin" or "auto", or for any config that
already writes a base_url line.

Fixed

Five kb keys verbs accepted --json, printed prose and exited 0. The flag is declared
once on the keys parser, because the verb is a positional, so argparse accepted it on all
seven. Only list and show build a JSON document. The other five wrote their ordinary log
lines and exited zero, so a script that asked for machine-readable output got prose with
nothing in the result to tell it apart from success. create, check, rotate, revoke and
prune now refuse the flag and name the two that honour it, exiting 2, the usage code, without
touching the keystore. The refusal goes to stderr, following the rule the two emitters already
keep: once --json is asked for, stdout carries the document and nothing else, so a refusal
cannot land inside a caller's > out.json.
kb serve printed a green success banner for a server that never started. A network
start that was about to refuse printed
```
✓ MCP server on http://127.0.0.1:8765/mcp  (Ctrl-C to stop)
  --keys-only refused: no key file with a live key was found ...
```
and exited 1 with nothing listening. The banner sat above the block that reads the key file,
so all five key-file refusals printed second, and an operator reading their terminal top to
bottom saw a running server. A code comment on that block claimed the opposite, that every
refusal ran before any banner.

The two banner lines are printed last now, right before the server call, with nothing
between them that can return. A start that refuses prints no banner at all. The refusals
covered: a key file that cannot be trusted, --keys-only with $CONTEXTLAKE_MCP_TOKEN set,
a file this reader cannot account for, a file whose keys have all expired or that holds no
records, and --keys-only with no live key anywhere. The --host refusal already ran before
the banner and is unchanged. Output on a start that goes ahead is unchanged.
kb serve --keys-file and --keys-only are documented. They decide which key file a
network start reads and whether it may mint a shared token, and docs/cli-reference.md
described neither. It carries the four-tier resolution order now, why an absent file at a
named path is refused rather than read as a first start, and the two cases --keys-only
refuses.
A first start named no way to stop using the shared token. Every key-file refusal told
the operator to run contextlake kb keys create <name>. The one route a new operator walks,
a first start on a machine with no key file, printed the unscoped token and named nothing, so
the person who most needed a scoped key was the only one never told the command exists. That
start now adds one line saying the token is shared and naming kb keys create. The
all-revoked start, which already named it for its own reason, is unchanged and does not say
it twice.
[serve] was reported as an unknown config table while the same run refused a start over
[serve] keys_file. One kb serve run printed config: unknown config table 'serve' (ignored) on stdout and Key file refused: [serve] keys_file names ... on stderr. The table
is known: keyfile._serve_keys_file opens the same TOML files and honours the value, which
is how the second line came to name it. The warning was the wrong line, and serve is in the
known-table set now. Keys inside the table are still not checked the way [kb] keys are.
The reason written down for one fail-closed refusal was false. A key file holding no
records is refused, and kb/keyfile.py justified that with "kb keys prune on a schedule
empties a file whose keys lapsed, so the calendar reaches this state too". Nothing schedules
prune; it runs when a person types it. The refusal is unchanged, because it never depended
on who emptied the file: a file holding no record admits nobody and reads on stderr like a
first start, so minting on it turns a deployment that asked for scoped keys into an open one.
The same false premise was restated in kb/cmds/serve.py's table of zero-live states and in
a test docstring; all three are corrected, and a test now fails if the sentence returns to
any of them.
kb enrich term selection: the cap is now spent on searchable symbols. The term builder
read repo_brief's top_symbols, which ranks every node in a repo and then caps the result,
and filtered that capped list down to the embeddable kinds. So files, packages, modules and
config keys consumed the budget before one searchable symbol was considered. Measured on 48
of a 56-repo store (the 8 largest shards left out to bound the measurement's own memory): 24
of the 29 repos holding 9 or more embeddable nodes got fewer than the 10 terms asked for, and
the largest of them, at 5,494 embeddable nodes, produced 5.

repo_brief now carries a second ranked list, top_embeddable_symbols, filtered to the
embeddable kinds first and capped after, one row per distinct name. The term builder reads
that. On the same 48 repos all 29 with enough symbols now reach 10 terms, and the total rises
from 191 terms to 321.

Your terms will change, not only grow. The count per repo rises or holds, and no repo ends
with fewer, but the ranking that fills the list is different, so the SET moves: 20 of the 32
repos that HELD a symbol term lose at least one they used to get, and 41 of the 143 previous
symbol terms are gone. A term drops out when a higher-degree definition displaces it.
(The denominator is 32, not the 48 measured: the other 16 held no symbol term to lose, so
counting them would understate the churn among repos this can affect.)

field and endpoint names can vanish from a repo's terms entirely. There is no per-kind
floor, so the ranking alone decides. On those 48 repos, field reached no term in any of the
4 repos holding field nodes, including one where 103 of its 178 embeddable nodes are fields;
endpoint reached none in 12 of the 15 repos holding one. Widening the budget barely helps:
re-measured at 25 terms a repo instead of 10, all 4 field repos stay empty, and endpoint
recovers in at most 2, leaving 10 of 15 still empty. So a bigger budget is not the fix for
either kind, and for field it changes nothing at all. This is deliberate. A repo gets 9 symbol names by default, across 19 embeddable kinds,
so a floor could reserve at most one slot each, and a per-kind floor is half of what starved
the old path. The full reasoning sits beside the ranking in wiki/generate.py.

top_symbols is unchanged, and so is every wiki page: its all-node candidate set is
deliberate, the new list stays out of the grounded_count/coverage_total ratio and out of
hubs/dispatchers, and no wiki, dashboard or MCP surface reads it. The 16 repos in that
measured set whose symbols were never extracted still get one term; that is indexer coverage,
and this change does not claim it.

Security

A config file found by walking up from the current directory may no longer choose an endpoint,
a credential variable, or a credential-carrying provider. The provenance gate covered the keys
that become a subprocess argv ([llm] command/args/provider = "cli", [[sources]]
command/args/mcp_command). It now also covers [llm]/[embeddings] base_url and
api_key_env, and [[sources]] mcp, token_env and auth_dir. [[sources]] scopes is
strengthen-only: a discovered file may narrow the OAuth grant and may not widen it.

On top of the per-key refusal, a discovered file may not aim a tier that carries a credential.
When the provider that wins the merge for [llm] or [embeddings] is openai or anthropic
and that value came from a discovered file, the tier is off for that run, rather than falling
back to built-in defaults nobody chose.

This breaks an honest project-local block that names openai or anthropic in a
.contextlake.kb.toml. Three things clear it: delete those keys from that file and set them in
~/.contextlake/kb.toml; pass --config PATH naming that file; or, for [llm] only, pass
--llm PROVIDER to kb wiki, kb docs or bootstrap. Adding the block to
~/.contextlake/kb.toml while the discovered file keeps its own provider line does not clear
it, because the discovered file is merged last and its provider still wins. Set
CONTEXTLAKE_NO_LOCAL_CONFIG=1 to skip ancestor discovery entirely.

Navigation Menu

contextlake 9.0.0

Choose a tag to compare

Sorry, something went wrong.

Sorry, something went wrong.

Uh oh!

No results found

Added

Changed

Fixed

Security

Uh oh!