Releases: sayak-sarkar/contextlake
Release list
contextlake 9.2.0
Added
-
A networked server now records which key called which tool, and
contextlake kb keys usagereads it back.kb keys list'sLAST USEDcolumn,
frozen at-since it shipped, is filled from the same file.Usage: 140 calls (140 timed) /home/you/.contextlake/kb/mcp-usage.jsonl KEY CALLS ERR THR DENY P50 P95 k_4f2a91 120 0 0 0 75ms 142ms k_9c01de 20 20 500 0 423ms 843ms Refused requests (never reached a tool) throttled 500 unknown 30 identity_unset 12 total 545A row has six fields and there is nowhere to put a seventh: the minute, the key id,
the tool name, the outcome, the tool time in whole milliseconds, and how many events the
row stands for. No query text, no symbol, no repository, no file path, no client address,
and nothing about the credential a refused caller presented. The recorder takes keyword
arguments only, with no free-text parameter and no**kwargs, so that is structural
rather than a sanitiser somebody has to remember to run.Counts, not lines, for traffic the server never admitted. The eight refusal outcomes
and the identity fault are counted into one row per key, tool and minute; 500 refused
requests are one line reading500. An unauthenticated flood would otherwise evict every
real row inside a minute. Calls from an issued key keep one row each, because a
percentile needs the individual values.A refused call is recorded too. The row is written in the tool wrapper's outer
finally, so a call refused by the tool grant, a call the rate limiter never admitted and
a call that raised are all in the file. Percentiles are nearest-rank; a refusal above the
concurrency slot carries no duration and prints-rather than0ms.Rows buffer in memory and are written every ten seconds and on shutdown, so a tool call
does no disk I/O. The file grows to 22,000 rows and is then trimmed back to the newest
20,000, so the rewrite happens once per 2,000 rows instead of once per append. A line the
reader cannot score, from a truncated write or a newer contextlake, is skipped and
counted, andkb keys usagesays how many rather than quietly reporting a short total.Three things it deliberately does not measure, each stated on the surface that prints it:
oneaskcounts once, asask, since it reaches its eight siblings below the wrapper;
tools/listand the handshake cross no wrapper, soCALLScounts tool calls and never
HTTP requests; andkb://statsresource reads are not recorded.Off with
--no-usageor[serve] usage = false.[serve] usage_max_linesand
usage_flush_secondstune it, read only from a config you named, the same gate
[serve] keys_fileand the quota defaults go through.stdio is unchanged, byte for byte. It builds no recorder, reads no ContextVar and does not
load the usage module at all. -
--rate,--burstand--cost-budgeton a key are now enforced over the
network. They were recorded and read by nothing. Measured on a live
kb serve --transport http --keys-onlyserver: a key created--rate 3/min --burst 4
answered four calls and thenHTTP/1.1 429 Too Many Requests content-type: application/json retry-after: 20 {"jsonrpc":"2.0","id":null,"error":{"code":-32000,"message":"rate limit exceeded for this key: 3/min. retry in 20s"}}Two buckets per key, filled lazily from two floats each.
--rateand--burstbound
requests;--cost-budgetbounds tool TIME, as a duration per period (30s/min), and
each call is charged how long its body ran. A duration rather than a count because a
count mispricesaskby 8x: it is one request and eight tool bodies, sinceask
reaches its siblings below the wrapper that could have counted them.The refusal is at the gate, before the request reaches any tool. So it costs one
header parse rather than a worker thread, and it coverstools/listand the
kb://statsresource, which cross no tool wrapper at all. A caller with no valid key
gets401and is never counted against a quota: identity resolves first, which is
what keeps the bucket map keyed by ids this server minted rather than by anything a
caller can forge.Values are validated now.
kb keys create --rate 60is refused at the flag, naming
the string, so a typo cannot be minted onto a key that then reads as limited. The same
parser runs over every stored value when a key file is loaded for serving: a bad value
exits 1 before the socket binds, and a bad value introduced by a live edit is rejected
with one warning while the previous keyring keeps serving.noneon any axis means no limit there, and beats a server default.--burstneeds
--rate: on its own it is the capacity of a bucket that does not exist. The minimum
burst is 4, because an MCP client spends three requests on the handshake before its
first tool call.Not persisted and not shared between processes: a restart refills every quota, and two
server processes give each key twice its quota. On thessetransport the 429 message
is lost and the session closes, which is a defect in that client, not in this server.
docs/mcp-transports.mdcarries all three.stdio is unchanged, byte for byte. It builds no limiter, opens no timer and does not
load the rate-limit module at all. -
[serve] default_rate,default_burstanddefault_cost_budgetinkb.toml, for
a quota that applies to every key that names none of its own. All three are unset out
of the box, so an upgrade starts limiting nobody. Read only from
~/.contextlake/kb.tomlor a file passed to--config: a.contextlake.kb.tomlfound
by walking up from the current directory is ignored with one line saying so, because a
rate limit a repository checkout can rewrite is not a limit.A shared token is bounded by
default_rateand has no per-credential opt-out, since it
has no key record to writenoneon. -
Unknown keys in
[serve]are warned about. The table was known but its keys were
never checked the way[kb]keys are, sodefault_rat = "60/min"was a silent way to
leave every key unlimited. It now prints one line naming the key and the known set. -
--toolsand--ownerson a key are now enforced over the network. They were
recorded and read by nothing. Measured on a livekb serve --transport http --keys-onlyserver: a key created--tools none --repos nothing-matches/*used to
get the full tool list and its calls all ran; the same key on the same server now
gets an empty tool list and a refusal that names the group which would grant the
call.--reposand--externalstill bind nothing and still print
(recorded, not enforced);--rate,--burstand--cost-budgetwent live in the
same release, below.Enforced at three surfaces, because a gate on one is a gate the caller walks around
by using another: the tool wrapper,tools/list, and thekb://statsresource, which
answers the countsgraph_statsanswers and crosses no wrapper at all.--toolstakes comma-separated groups (graph,search,docs,stats,owners,
semantic) plusall,readandnone.readis every group exceptsemantic. A
group this server does not know is refused atcreate, so a typo cannot be minted
onto a key that then reads as scoped. In a hand-edited key file the same value is
denied rather than refused: it narrows the key and never widens it.askis refused unless every tool it routes to is granted. It calls eight siblings
directly, below the wrapper that checks a grant, so a key grantedaskand denied
blast_radiuswould otherwise reachblast_radiusthrough the impact route.--owners realallowswho_knows;pseudonymousandhiddenrefuse it, and refuse
askwith it. There is no anonymiser on the network path, so a key that asked for
pseudonyms gets no names rather than real ones.--reposis deliberately not enforced. It cannot be decided from a call alone: a
node id does not carry the repository it came from, andrepo_dependencies,
repo_flowandrepo_event_flowtake a requiredrepoand return rows naming other
repositories. Correct scoping needs a filter inside the store.stdio is unchanged, byte for byte. It reads no key, no policy and no identity, and it
loads no grant module.
Changed
-
The
(recorded, not enforced)label moved from per line to per axis. One label
after all three scope axes claimed the same thing about all three, so enforcing
toolsalone would have made the line sayreposandownerswere live too.
kb keys shownow marks each axis on its own, and an unset axis carries no marker at
all, because it records no scope for a marker to qualify. -
kb keys showandlistprint the EFFECTIVE quota and where each value came
from. The limits line used to read a bareunsetfor a key that named no rate. With
[serve] default_rateset, such a key is limited, and an operator readingunset
hands it out believing it is not. It now renders one of four states per axis:
rate=60/min (enforced),rate=unset -> 60/min from [serve] default_rate (enforced),
rate=unset (no limit), orrate=none (enforced: no limit, set on the key). The
ratecolumn inlistshows the effective value for the same reason.Each per-record
--jsondocument gainseffective_rate,effective_burst,
effective_cost_budget(strings or null) andlimits_source, an object mapping each
of the three axes tokey,configorunset. No field is removed and none changes
type. -
policy_enforcedin every--jsondocument is derived rather than a fixed
false. It answers whether every axis the document renders is enforced, so a key
scoped only on--toolsreads `t...
contextlake 9.1.0
Added
-
--jsonon all sevenkb keysverbs.create,revoke,rotate,pruneand
checknow emit a document, joininglistandshow.9.0.0 made those five refuse the flag at exit 2. That was the honest interim state: the
release before it let them take--json, print their ordinary log lines and exit 0, so a
script that asked for machine-readable output got prose and no error. Refusing was better
than lying about it, and answering is better than refusing.Standard output carries the document and nothing else, on every exit path. A failure is a
document too, carrying"error"with a snake_case code, which is whatkb query,
kb owners,kb impactandkb evalalready do.Three fields exist because an exit code could not carry the answer:
changedonrevoke,rotate,pruneandcreatesays whether the key file was
written. Revoking a key somebody else already revoked exits 0 and changes nothing,
which read the same to a script as revoking it.reasononcheckis one ofmalformed,unknown,revokedorexpired. All four
exit 1, so a CI gate that warns on one and fails on another had nothing to read.last_used_stateisnot-recordedin this release.last_used_atisnullfor two
different reasons and the sibling is the only thing that separates them.
create --jsonandrotate --jsonkeep the key on stderr and report
"key_shown_on": "stderr".--json > out.jsonwould otherwise write a live credential
into a file at the caller's umask.--print-keymoves the key into the document'skey
field, and it already refuses a terminal. -
rotatehonours--print-keyand--out. Both flags parsed onrotateand both
were ignored, exiting 0. The new key exists nowhere else, so a rotation script had no
route to it but scraping stderr.
Fixed
-
kb keys create --out <existing path>minted a key and lost it. The record was
written to the key file, and only then was the--outpath refused for already existing. The
command exited 2 saying nothing had worked while a live record sat in the file whose
plaintext had never been shown to anybody. The output file is now opened before the key
is minted, and removed again if the mint fails.If you ran that command on 9.0.0, look at
kb keys list. An orphaned record is
indistinguishable from a key you hold: samelivestate, same emptyLAST USED, and no
field on the record says whether the key ever reached anybody. Find the records matching
the names you tried to create, andkb keys revokethem. Each retry of the failing
command minted another one, so there may be more than one per name. -
kb keys show --jsonon an unknown id emitted no JSON. The not-found branch ran
ahead of the--jsoncheck, so it printed prose to stdout and exited 1. It now emits
{"error": "unknown_id", ...}at the same exit code, as dorevokeandrotate. -
--overlaphelp said "default 0". The default is7d.
contextlake 9.0.0
Added
-
contextlake kb keys, the command every key-file refusal already named. Seven verbs:
create,list,show,revoke,rotate,checkandprune.kb serverefuses to start on several key-file states, and each refusal told the operator
to runcontextlake kb keys create <name>. That command did not exist. A server that
refuses and names a way out the reader cannot take is worse than one that starts wrongly,
because the operator has nothing to do next. This is that command.The key is shown once. It goes to standard error at creation and never appears again,
because the file stores a SHA-256 digest rather than the key. It is deliberately never sent
through the logger: the console handler always writes to stdout, a--log-filerun adds a
5 MB rotating file with three backups that outlives the process, and the redactor rewrites
workspace paths and repo names, so it would scrub a key on neither. A lost key is rotated,
never recovered, androtatekeeps the old key working for--overlapso the holder can
swap without an outage.Two other paths can carry the key out, and both are narrower than they look.
--print-key
writes the bare key to stdout for a pipe and refuses a terminal, where it would land in the
scrollback instead of a secret store.--out FILEwrites it at mode 0600, with the mode set
at creation rather than chmod-ed afterwards, and withO_EXCLso an existing path is
refused rather than overwritten.checkreads the key from standard input only. A key on a command line lands in shell
history and shows inpsto every account on the machine. A terminal with nothing piped
in is refused too, rather than waiting for end-of-file behind a blank screen: it does not
prompt, because a typed key lands in the scrollback. It opens no socket and sends no
request, which is what lets it answer when the server is the thing that is down, and it
says so rather than implying it verified anything against a server.No verb opens the store database. Every one of them runs on a machine that has never
built an index, which is the machine an operator is on when a server has just refused to
start.kb keys listis the first command they run.Two refusals split by verb rather than collapsed into one rule. A key file carrying group
or other bits, or sitting in a directory anyone can write to, is a policy fault: write
verbs refuse, andlistwarns and prints the table anyway, because blocking the operator
from seeing what exists is the wrong failure whenlistis how they diagnose the refusal
they just hit. A file that cannot be read at all is not that: nothing was read, so every
verb fails and names the path.The scope flags are recorded and enforced by nothing, and every surface says so.
--tools,--repos,--owners,--rate,--burstand--cost-budgetare written onto
the key and rendered back bycreate,list,showandcheck. No code reads them. A
key created with--tools none --repos nothing-matches/*was presented to a live
kb serve --transport http --keys-onlyserver andtools/listanswered with all 23
registered tools, one of which then ran and returned a result.So the values print with
(recorded, not enforced)beside them and three lines saying
what that means,show --jsonandlist --jsoncarry"policy_enforced": false, and an
unset axis readsunsetrather thannone, which read as a denial for the key a bare
createmakes. Telling an operator their key is scoped when it is not is worse than not
offering the flags, because they hand the key out on that reading.Scope is per tool, not per repository once it does work: a key allowed a tool will read
every indexed repo through it.--rateand--cost-budgetare stored as typed and
validated by nobody, because their parser ships with the rate limiter. TheLAST USED
column readsneveruntil the usage file it reads from exists. -
Per-request identity on the network MCP transports, and the frame that will carry access
control. Groundwork only: nothing is enforced yet and nothing changes for a local run.build_http_appnow resolves aPrincipalper request and the tool wrapper reads it, so
two callers holding two different credentials are two different identities inside a tool
body rather than one anonymous caller. Until now there was no place to put that fact, which
is why access control, rate limiting and usage accounting could not be built: each would
have had to invent its own notion of who was asking.The wrapper's whole
try/finally/exceptstructure lands here, once, deliberately. Four
planned stories each need to add a line to that twelve-line function, and when they were
specified separately their orderings contradicted each other. Landing the frame first turns
each of them into an insertion at a named anchor instead of a restructure, so whoever lands
second does not have to unpick whoever landed first.It fails closed. Whether identity is required is a build-time decision made by
build_http_app, never inferred from whether an identity happens to be present. A stdio run
does not read the value at all, and on a network run a missing identity refuses the call
rather than answering it unscoped. Those two states used to be the same value, which meant a
server whose identity plumbing broke would have answered every request as if unauthenticated
access were intended, with every test still passing.What it does not do. It detects a MISSING identity. It cannot tell a WRONG one, and on
the SSE transport the plausible failure is substitution rather than absence. Closing that
needs a per-connection token compared at the boundary, which is specified and not built.
askcalls its sibling tools directly, so those legs never cross the wrapper and an access
check placed there will not cover them; the anchor comment says so. And the run outcome the
wrapper records has no reader until the usage recorder lands.stdio, the default, is unchanged: same tool output, no identity lookup, no new file,
config field or dependency for anyone who never serves over the network. -
contextlake kb source wizard. It lists every configured source with a reachability
mark, then offers to add another and loops until you answer no (pressing enter is no). The
survey reads the sameverify_sourcepathkb doctorandkb source testuse, so
"is this source reachable" has one answer across all three. The add step iskb source add
run interactively, so the prompts, the literal-secret refusal and the write target are the
same ones. It needs a terminal: a prompt written to a pipe hangs, so a non-interactive run
is refused with exit 2 and the flag form to use instead.
Changed
-
kb enrichreports edges to code, not only documents stored. The linking step already
ran: every enrichment document was matched against the repo's symbol names and the matches
were stored asdocumented_byedges. The count was then discarded, so the run could report
only how many documents came back. A document with no edge to any symbol cannot answer a
question about the code, and it read as a success.The run now prints, per repo, the terms tried, the documents returned and the edges attached
to code, and closes with a line that puts every targeted repo in one of five buckets:
enriched, nothing returned, returned but unattached, failed, skipped. The five add up to the
number of repos the run planned to touch, on every exit path."Returned but unattached" is a state, not a failure. The matcher is whole-word with a
three-character floor, so a ticket that discusses a repo in prose without naming a symbol
correctly attaches to nothing. That run still prints✓.A repo whose store or shard write fails is now counted and reported instead of aborting the
whole run. A run where every repo failed that way exits 1.API change:
run_enrich_reporeturns anEnrichCounts(terms, documents, edges)triple
instead of the document count alone. -
kb doctor's per-source line now separates three answers that used to render as one
⚠. A source that was dialled and did not answer keeps⚠. A source of a type with no
reachability probe (gitlab,zendesk) now draws⊘, matching whatkb source test
already prints for the same case. A source withenabled = falsealso draws⊘and is not
dialled at all, matchingkb connectandkb ingest, which both skip disabled sources.
None of the three changes doctor's exit code, which is unchanged and still deliberate.What this loses: doctor no longer reports a broken path or an unreachable endpoint on a
disabled source. Nothing reads that source, so the round trip bought nothing, but the line
used to be there. Re-enable the source to have it dialled again. -
The bundled
--sampledemo fleet moved to a new domain, and two of its repos changed
shape. Repo ids and symbol names changed, so a script that names one has to be edited. The
old fleet's domain read as a real production estate rather than as an obvious invention, which
is what demo data has to be. It now models a weather-station monitoring network: stations
report readings, a forecast service runs a model over them, an ingest pipeline normalises raw
readings, an alerts service fans out severe-weather notices, and a console UI shows it.acme/auth-service -> acme/station-registry acme/catalog-api -> acme/forecast-api (rebuilt, not renamed; see below) acme/payments-api -> acme/sensor-ingest (call edges redirected; see below) acme/web-ui -> acme/console-ui acme/notifications -> acme/alerts acme/shared-lib -> acme/shared-lib (unchanged) demo/app -> demo/app (id unchanged; its two symbols...
contextlake 8.13.0
Changed
-
The ambiguity fanout cap is 10, not 6, and stores re-index to pick it up. A call
reference naming a symbol defined in more than the cap's many places produces no edge at
all, so the caller is simply absent from "who calls X". Measured with the cap removed on
the two largest ambiguity contributors in a 717,381-node store, the old cap was dropping
29.6% and 37.0% of resolvable call references. The comment beside it claimed 21.6%
and "no knee in the distribution"; both were wrong. There is a knee, and 10 sits on it:
76.2% and 80.6% of references for 1.22x and 1.52x the ambiguous edges.Admitting everything is still wrong, for a sharper reason than the old note gave. The
uncapped cost is one or two pathological names per repository: 2,864 sites naming a
symbol with 1,432 definitions produced 4.1M edges by themselves, 92% of that
repository's uncapped total.PARSER_VERSIONmoves to12, so existing stores re-index rather than keeping edges
built under the old cap.
Added
-
Zoom in and zoom out buttons on the graph toolbar. Zoom level could only be chosen
by wheel or pinch. Pinch is a multipoint gesture, and WCAG 2.5.1 (Level A) wants a
single-pointer alternative; "Fit to view" sets a zoom level but does not let you pick
one. Each press steps by a fixed ratio about the viewport centre.Wheel zoom got more responsive alongside them:
wheelSensitivitywas0.2, so reaching
a readable scale took a dozen notches and the canvas read as stuck. It is now1.
Fixed
-
Every graph and dashboard page requested a favicon that did not exist, producing a
404 in the console on each load. Both now carry an inline SVG icon, which costs no
request and survives an offline export. -
kb dashboard --site <dir>overwrote files it did not write. The export writes
index.htmlunconditionally, so pointing it at a directory holding anything else --
a docs site, a hand-maintained landing page -- replaced that content silently, with the
command reporting success. It now refuses any directory containing files the export does
not itself produce, naming what it found. Re-running into a previous export stays
idempotent and needs no flag.
contextlake 8.12.0
Added
-
A node in the architecture graph now opens the wiki for its repo, and lands on the
subsystem page covering that node's file. The graph page had no wiki link of any kind.
Inside the dashboard the graph runs in an iframe, so the node's control asks the
dashboard to route; the dashboard resolves the file to the narrowest generated
subsystem page that contains it and opens that, falling back to the repo's own page
when no subsystem page covers it. Opened through the dashboard's Fullscreen link the
same page has no parent, so it links that route directly. A static export links the
sibling wiki page it already writes.The control appears only for a repo that has a generated wiki. The prefix match is
anchored on a path segment, sosrccannot claimsrcutil/helper.py.There is no jump to a heading, and that is a limit of the data rather than an
omission. Generated pages carry page-level headings only (Overview, Setup & Run,
Architecture, Dependencies, Gotchas), the model is told to omit any it has nothing to
say for, and nothing in a page is about a single symbol. A computed anchor would land
silently at the top of the page. -
The docs pages that explain the graph now carry the running graph.
asking-the-graph,code-graph-model,indexing-the-code-graphand
visualizing-the-graphdescribed the visualizer in prose while the live page sat one
directory away, reachable only from the landing page. The page is 788 KB, so it is not
embedded eagerly: the markup ships a screenshot and an IntersectionObserver swaps the
iframe in when the reader scrolls near it, carrying the current theme. A reader with no
JavaScript, or who never scrolls that far, keeps the screenshot.
Fixed
-
A static export's graph pages disagreed about which repos had a wiki. The map was
filled while the pages were being written, so each page saw only the repos written
before it and the fleet overview, written first, saw none. Nothing read the map yet, so
nothing failed. It is now built before the first page. -
RepoTooLargetold you to pass a flag that does not exist. A repository over the
memory budget was refused with "narrow it with--languages". There has never been a
--languagesflag; the message introduced it, so anyone who followed the advice got
"unrecognized arguments" and no way to act on the error. The setting is real and lives
inkb.tomlaskb.languages, which the message now names. A guard was added: no
string inside araiseor an exception's__init__may name a long flag that no parser
registers. -
The command palette's search field removed its own focus ring.
.cl-palette__input:focussetoutline: none, which takes the indicator away from
keyboard users rather than only from mouse users. It was safe while the palette held one
focusable control and would have become a WCAG 2.4.7 failure the moment a second one was
added, silently. Narrowed to:focus-visible, with a stylesheet-wide guard.
contextlake 8.11.0
Fixed
-
The dashboard rendered twice on every load, and fetched
/api/overviewtwice with it.
boot()attaches ahashchangelistener and then gives the page a default hash when it
opens without one. Assigninglocation.hashfireshashchange, and the listener is
already attached, so the page rendered once from the explicit call and once from the
event. The two requests overlap, so neither can serve the other from cache. Measured
against a 961,633-node store: 2,769 ms and 4,005 ms, 35 ms apart, while the page shell was
ready in 114 ms.history.replaceStatewrites the same URL and dispatches nothing. It
also keeps the default hash out of the history stack, so Back no longer returns to the
hash-less URL and straight back again. -
The dashboard reserved layout columns for a sidebar and drawer that were not there.
The rail becomesposition: fixedbelow 768px and the drawer below 1280px, so both leave
the grid, but the rules naming their columns applied at every width. Specificity is
resolved before any media query is considered, so those rules won regardless of source
order. On a 700px viewport the main content measured 64px wide with the rail collapsed.
Three of the four rail and drawer combinations were wrong, not the one that was reported.
All four now use the full width, and 1000px and 1400px were re-checked. -
kb wikiunder-reported how much of a run did not happen. When a repository's
whole-repo page fails, the run skips that repository's module pages rather than trying
each one. Those pages were counted nowhere, so the four totals added up to less than the
run planned and six missing subsystem pages read the same as a repository that had none.
They are now reported as "N not attempted", kept separate from failures because a page
nobody tried is a different fact from a page that broke. The all-failed line also called a
page count "repo(s)", so a run that lost three module pages of one repository announced it
had failed for three repositories.
Added
-
A truncated graph view now names the node kinds it dropped. "500 of 3,200" says a view
is partial without saying that the part you came for is the part that is missing: a
repository with 412tablenodes that renders none of them has an empty ER diagram, and
the old message could not tell that apart from dropping 412 low-value nodes. The four
worst losses are logged with the count each kind had, and the full breakdown reaches
callers asdropped_by_kind. Both sides are counted rather than estimated. The key is
absent on a complete view, so nothing can render "0 dropped" over a view that dropped
nothing. -
A
forced-colorsblock for the dashboard, where there were none. Most of it needed
nothing: the health chips carry the words Fresh and Stale, the confidence chips carry a
border style and a clipped glyph, and cards, the rail and the drawer have borders that
survive a forced palette. The trust bar does not. Its segments are sized by flex and told
apart by background colour alone, so a forced palette merged them into one bar; they now
carry a divider, and focus uses the system highlight colour.
Documentation
-
The
--repospattern syntax is documented in full, in
Mirroring repositories. The page said patterns are
globs and are anchored; it did not say which wildcards are available. It now lists all
four (*,?,[abc],[!abc]) with an example each, and states the four rules that
govern every pattern: comma-separated, anchored, case-insensitive, and matched against
both the group-qualified and the local path.It also covers the one case the anchoring does not reach. There is no escape character,
soodd*namematches a repo literally namedodd*nameand matchesoddXnameas well,
and cannot select the first on its own. A one-character set does:
--repos "odd[*]name".*and?are legal in a path on Linux and macOS, not on
Windows.Every claim on that page is now pinned by
tests/test_repos_pattern_syntax.py.
contextlake 8.10.1
Fixed
-
kb index --workspaceno longer breaks its worker pool. The cause was
RepoTooLarge, the exception 8.10.0 added for the memory budget: it could not be
unpickled. A worker that refuses a repository sends the exception back to the parent,
and Python rebuilds an exception ascls(*args), whereargsheld only the formatted
message this class passes toException.__init__. The rebuild was three arguments
short and raisedTypeErrorinsideProcessPoolExecutor's manager thread, which has
no future to attribute a failure to, so it broke the whole executor: every healthy
worker was sent SIGTERM and every pending repository failed withA process in the process pool was terminated abruptly. One refused repository ended a 656-repository
run in about a minute.pickle.dumpssucceeded on the exception throughout. Only the parent's unpickle
failed, which is why nothing caught it. The same 17-repository cluster that broke the
pool twice now completes at--workers 4with 0 failures, the over-budget repository
reported as a skip, and all four workers exiting 0. -
Five more exceptions had the same defect and are fixed with it.
GrammarNotInstalledis raised on the same indexing path and would break a pool the
same way when an optional grammar is absent.McpToolError,CircuitOpenError,
StoreBusyandRunBusyare off that path today. Every exception in the package that
defines its own__init__now defines__reduce__, and a test walks the package to
assert each one survives a round trip, so the next one cannot ship without a sample. -
A broken worker pool now falls back to serial indexing. That fallback was written
for this failure and was unreachable:BrokenProcessPoolsubclassesRuntimeError, so
the per-repositoryexcept Exceptioncaught it, counted one failure against whichever
repository raised it, and continued, and the handler that re-runs the work-list serially
never ran once. The failure counters are reset before the serial pass, so a repository
that genuinely failed before the break is not counted again by it. -
The reason a pool broke is now reported.
str(BrokenProcessPool)is a fixed
sentence naming no cause, and the real reason is attached as__cause__, which was
discarded. Reporting only the fixed sentence is why three candidate causes stayed
unseparated across several investigations.
contextlake 8.10.0
Upgrading
This release re-indexes every repository on your next kb index. PARSER_VERSION moves
from 10 to 11 because .config and four sibling extensions now carry settings into the graph,
and no commit moves when an extractor starts reading a file type it previously ignored. Without
the bump an existing store would report every repository "unchanged" and never gain a single
setting.
What that costs, measured on a 660-repository fleet:
- The rebuild is automatic.
kb indexreportsolder parser (10 -> 11)and re-indexes rather
than skipping; the pass after that is quiet again. - The new extensions add roughly 61,000
config_keynodes fleet-wide, dominated by 1,023
.configfiles at about 57 settings each. - The store-size change was not measured and no figure is given for it here. A full
re-index of that fleet has not been completed on this machine, and a number nobody measured
is worse than an absent one.
Known issues
kb index --workspace can break its worker pool at --workers above 1. Reproduced twice
on a 17-repository cluster: a worker dies immediately after a large shard completes, and every
still-pending repository fails with A process in the process pool was terminated abruptly. On
a 656-repository run this failed 640 of them in about a minute.
--workers 1is verified working on the same cluster, including every repository that
breaks the pool.- 2 through 8 are untested. The default is
min(8, cpu_count - 1), so a plain
kb index --workspaceis affected. - It is not a simple out-of-memory: one reproduction broke with 8.9 GB free, another with
4.3 GB, and system memory dropped at the instant of the break, which is one worker dying
rather than the machine starving. - The root cause is not established. The candidates not yet separated are a transient
out-of-memory kill of a single worker, a failure transferring a very large shard back to the
parent, and a grammar crash.
The max_repo_memory budget added in this release does not prevent it, and the reason is the
limit that budget documents: it is linear in input bytes, while the cost here tracks edge
count. One of the repositories involved estimates 0.86 GB and produces 166,000 edges.
Added
-
.config,.props,.targets,.settingsand.plistnow reach the XML
config extractor, andPARSER_VERSIONmoves to 11 so existing stores get them.
Without the bump the extraction reaches nobody who already has a store:kb index
gates re-indexing on the parser stamp, so an already-indexed repository reports
"unchanged" and never gains a setting. Re-indexing happens automatically on the
nextkb index. Only.xmlreached the extractor before, so the canonical .NET
settings file was contributing nothing: measured across 660 repositories, 1,023.configfiles
produced zero nodes while being exactly the files "where is this setting
defined" is asked about..resxis deliberately still excluded, being
localisation rather than settings and worth roughly 91,000 nodes fleet-wide;
so are project files, which the manifest extractor owns, and.svg, which is
XML-shaped graphics. -
[kb] max_repo_memory, a per-repository memory budget checked before any file
is parsed.max_file_bytesbounds one file and cannot bound a repository that
is wide rather than deep. The repository that took a 15.4 GB machine down had a
largest file of 3.57 MB against a 5 MB cap, so that cap never fired once, while
1,432 XML files averaging 0.42 MB added up to 671 MB. The new budget estimates a
repository's cost from a stat-only pass, weighting each file kind by measured
peak memory per byte (code 19.6x, SQL 5.0x, XSD 4.3x, XML 3.5x), and skips the
repository with its name and the dominant kinds if it would exceed the budget.
It defaults to 3 GB, taken from the fleet rather than chosen: across 660 real
repositories the median estimate is near zero and p99 is 1.69 GB, with three
outliers at 6.09, 6.76 and 7.35 GB. Set it to 0 to disable. The estimate is
linear while the real cost is not, so it runs low on the largest repositories;
it is a coarse guard, and the existing shard-item check remains the second layer.
contextlake 8.9.0
Added
-
AWS and Azure adapters.
--platform awscreates an EventBridge Scheduler
schedule firing an ECS task;--platform azurecreates a Container Apps Job on
a cron trigger. Both shell out to an already-authenticatedawsoraz, so
contextlake still ships no cloud SDK. On EKS and AKS use--platform k8s
instead: both are Kubernetes, so theCronJobadapter serves them and brings
concurrencyPolicy: Forbidwith it. Neither cloud service has an equivalent of
Forbid, so two runs there can overlap and the second skips on the store's
advisory lock rather than never starting, which both adapters report. They round
differently by design: EventBridge takesrate(N minutes)and rounds to whole
minutes, while a Container Apps Job trigger is a cron expression and rounds the
way cron does. Registered but never auto-detected, for the same reason as the
Kubernetes adapter. Verified by asserting the rendered request documents and the
exact CLI arguments: there is no account here, so neither is verified by
execution. -
A Kubernetes adapter, covering OpenShift as well. Renders a
CronJoband
applies it withkubectl, falling back tooc. One adapter serves both,
because OpenShift is Kubernetes with a stricter default security context and
the manifest satisfies the stricter one: norunAsUser, since the restricted
SCC assigns an arbitrary UID and rejects a pinned one, plusrunAsNonRoot,
a dropped capability set and anfsGroupso the mounted state directory stays
writable.concurrencyPolicy: Forbidgives single-writer semantics from the
cluster, so the second of two overlapping runs never starts. State mounts a
PersistentVolumeClaimrather than anemptyDir, because an ephemeral store
re-indexes the whole fleet every run. The schedule is a cron expression and
rounds through the same function the cron adapter uses. Nothing is patched in
the cluster on its own: changing an interval means re-running
schedule install, since a background rewrite would need cluster-write rights
for the life of the schedule. Reachable with--platform k8sand never
auto-detected:kubectlon a PATH does not mean a schedule belongs in that
cluster. -
A Windows adapter, so
contextlake scheduleworks through Task Scheduler.
Creates a task withschtasks /SC MINUTE /MO nunder a\contextlakefolder.
Two limits are reported rather than hidden./MOcounts whole minutes, so an
interval is rounded the way cron's is, down above a minute and up below it, and
installsays when it rounded.schtaskscannot set StartWhenAvailable, so a
run missed while the machine was off is lost; the adapter reports that with the
same phrase cron uses, which is what stopsstatusprinting the fact twice. The
command is quoted with Windows rules rather than POSIX ones, because a venv path
containing a space is the ordinary case there. Verified by asserting the exact
schtasksarguments: the development machine is Linux, so this backend is not
verified by execution. -
A launchd adapter, so
contextlake scheduleworks on macOS. Renders a
LaunchAgent plist withStartIntervalin seconds, installs it with
launchctl bootstrap gui/$UID(not the deprecatedload, which can return 0
while doing nothing), and reads the interval back off the installed plist
rather than reporting what was requested. launchd replays a run missed while
the machine was asleep, like systemd and unlike cron.schedule statusreports
no next-fire time for it, because launchd exposes none for an interval agent
and a computed guess would drift from what it actually does. Verified by
asserting the rendered plist and the exactlaunchctlarguments: the
development machine is Linux, so this backend is not verified by execution. -
schedule listreports units whose job record is gone.state()can only
answer "is job X installed?", which can only be asked about a job that still
has a record. The reverse had no reader: delete a record and its unit keeps
firing on schedule, is absent fromlist, anduninstallcannot reach it,
because it resolves a job name through the record that is gone. Adapters gain
installed_names(), implemented by reading the unit directory for systemd and
the marked crontab blocks for cron.listnames each orphan and its platform,
and says how to remove it. It also names any platform it could not enumerate:
skipping a platform and finding nothing on it both produce an empty result, so
reporting only the empty one would let "never looked" read as "checked, clean". With
--jsonthe two arrive as_orphaned_unitsand_unchecked_platforms, added
alongside the jobs rather than nesting them: a script reading this output keeps
working. The leading underscore is what makes that safe, since a job name must
start with an alphanumeric and so can never collide. -
schedule recommendsays when the activity bound was never measured. The
freshness half of the interval formula needs a count of how many repositories
changed, which only the index stage records. On an install without thekb
extra nothing records it, so the bound never engages and the interval rests on
the duty-cycle floor alone. Theactivity floorline was omitted entirely in
that case, which read the same as the bound being switched off. It now states
that it was not measured and what records it.--jsongains anactivity
field readingnot-measured,no-changeormeasured:floor_activity_seconds
is null for the first two of those, so the number alone could not tell them
apart. -
A memory-budget guard on
kb index. One repository in a real fleet
needed more than 9.3 GB in a single worker and never finished; no worker
count survives a repository that size. A repository whose parsed shard
exceeds 2,000,000 combined nodes and edges (well above the largest
repository known to index successfully, at roughly 356,000) is now skipped
with a named, explicit log line instead of being persisted, and the run
continues with the rest. This is a guard, not a fix: it stops one
pathological repository from taking a whole run down, it does not make that
repository indexable, and the exit code reflects that the run was not fully
clean. -
--workers Nonkb indexandbootstrap. Caps how many repositories the
index stage parses in parallel._index_workspacealready accepted and honoured
aworkersvalue, but nothing wired a flag to it, so the default (one fewer than
the CPU count, capped at 8) could not be overridden anywhere, including on
bootstrap, the default scheduled job. Lower it to cut peak memory on a large
fleet or a small machine.
Fixed
-
Every scheduled job read every other job's run history. All jobs append to
one history file and nothing in a record said which job wrote it. Two things
followed.decide_kindasks whether a successful full rebuild is older than
schedule_full_every, so a rebuild run by one job answered that question for
a job that had never run one and postponed its rebuild by a whole cycle. The
recommender's median run duration mixed every job's durations, so a
two-minutekb indexand a forty-minutebootstrapproduced one interval
that fitted neither. Records now carry the job name, passed to the child in
CONTEXTLAKE_SCHEDULE_JOB, and reads are scoped to one job. Records written
before this carry no job name and count as the default job's, so no existing
install loses the measurements it has earned. If you created a named job with
schedule intervalon 8.8.0, its earlier records are reattributed to
default, so that job starts from an empty history and runs one extra full
rebuild on its next cycle. Every record it writes after that is tagged. -
kb index's parallel path leaked every completed repository's parsed graph
for the whole run. The worker pool'sfutsdict was keyed byFuture, and
fut.result()does not clear a future's cached result, so a completed
repository'sGraphShardstayed reachable throughfutsuntil the pool's
withblock exited. Measured by A/B on the same 45 repositories at the same
worker count: 2,130 MB retained versus 95 MB released, 22x apart on
identical work, growing with repos indexed rather than with worker count.
Each future's dict entry is now dropped once its(repo_id, path, head)is
read off it, both on the fast path and the serial fallback after a broken
pool. -
A timed-out scheduled run orphaned its worker pool.
contextlake schedule run
spawned its child withsubprocess.run(..., timeout=...), which on a timeout kills
only the direct child. The child is usuallybootstraporkb index, which runs a
ProcessPoolExecutorof up to 8 workers, so a timed-out run left the whole pool
running, reparented to init and still holding memory. Measured on a real machine:
one killed run left 8 orphaned workers holding 12.4 GB. On a schedule, every
timed-out run leaked another pool. The child now starts in its own process group
and, on a timeout, the whole group is signalled (SIGTERM, then SIGKILL if it does
not exit within a few seconds), so nothing below it survives. Windows has neither process
groups norSIGKILL, so there the timeout falls back totaskkill /F /T, which
walks the child tree and reclaims the pool the same way.
contextlake 8.8.0
Added
-
contextlake schedule: a self-scheduler. Measures how long a run takes and how often your
repositories change, works out an interval from the two, and installs a systemd user timer or
a crontab entry that keeps the mirror and the knowledge layer current on its own, with no unit
file or cron line to write by hand. Core tier: works without the[kb]extra.
recommend,statusandlistread only;install,interval,resetanduninstall
write. Ad-hoc jobs run any contextlake command on their own interval:contextlake schedule interval 6h run -- kb wiki --force
Ten
schedule_*config keys, an honest split between what systemd and cron can each do, and a
container refusal for state that will not survive a restart. Discarding measured history
(--purge,reset --history) renames the history file to a.discardedsidecar instead of
deleting it, so a mistaken discard is recoverable. See Scheduling runs. -
branch_map: per-repository branch pins.branchputs the whole fleet on one branch,
which is the common case and not the only one.branch_maptakes comma-separated
pattern=branchpairs using the same globs asrepo_filter, so one team can track
developwhile a legacy tree sits onmaintenanceand everything else follows the
most-active selection. First match wins, so a specific entry can precede the glob that
would otherwise catch it. It beatsbranch, falls back to it, then to the selection.
A repository whose mapped branch does not exist is reported asunpinned, exactly as
branchalready does, rather than being switched to something else.branch_map = team/api=develop, legacy-*=maintenance
Changed
-
bootstrapgained--force. Rebuilds everything instead of only what changed: every
repository re-parsed, every node re-embedded. It did not exist as a flag before; it is what
schedule's periodic full cycle runs. -
configuration.md's example config now shows the scoping keys.repo_filter,branch
andbranch_mapwere documented in the settings table but absent from the example anyone
copies, so the two that already existed were easy to miss.