Skip to content

Releases: sayak-sarkar/contextlake

contextlake 9.2.0

Choose a tag to compare

@github-actions github-actions released this 06 Sep 23:39

Added

  • A networked server now records which key called which tool, and
    contextlake kb keys usage reads it back.
    kb keys list's LAST USED column,
    frozen at - since it shipped, is filled from the same file.

    Usage: 140 calls (140 timed)  /home/you/.contextlake/kb/mcp-usage.jsonl
    
    KEY       CALLS  ERR  THR  DENY    P50    P95
    k_4f2a91    120    0    0     0   75ms  142ms
    k_9c01de     20   20  500     0  423ms  843ms
    
    Refused requests (never reached a tool)
      throttled       500
      unknown          30
      identity_unset   12
      total           545
    

    A row has six fields and there is nowhere to put a seventh: the minute, the key id,
    the tool name, the outcome, the tool time in whole milliseconds, and how many events the
    row stands for. No query text, no symbol, no repository, no file path, no client address,
    and nothing about the credential a refused caller presented. The recorder takes keyword
    arguments only, with no free-text parameter and no **kwargs, so that is structural
    rather than a sanitiser somebody has to remember to run.

    Counts, not lines, for traffic the server never admitted. The eight refusal outcomes
    and the identity fault are counted into one row per key, tool and minute; 500 refused
    requests are one line reading 500. An unauthenticated flood would otherwise evict every
    real row inside a minute. Calls from an issued key keep one row each, because a
    percentile needs the individual values.

    A refused call is recorded too. The row is written in the tool wrapper's outer
    finally, so a call refused by the tool grant, a call the rate limiter never admitted and
    a call that raised are all in the file. Percentiles are nearest-rank; a refusal above the
    concurrency slot carries no duration and prints - rather than 0ms.

    Rows buffer in memory and are written every ten seconds and on shutdown, so a tool call
    does no disk I/O. The file grows to 22,000 rows and is then trimmed back to the newest
    20,000, so the rewrite happens once per 2,000 rows instead of once per append. A line the
    reader cannot score, from a truncated write or a newer contextlake, is skipped and
    counted, and kb keys usage says how many rather than quietly reporting a short total.

    Three things it deliberately does not measure, each stated on the surface that prints it:
    one ask counts once, as ask, since it reaches its eight siblings below the wrapper;
    tools/list and the handshake cross no wrapper, so CALLS counts tool calls and never
    HTTP requests; and kb://stats resource reads are not recorded.

    Off with --no-usage or [serve] usage = false. [serve] usage_max_lines and
    usage_flush_seconds tune it, read only from a config you named, the same gate
    [serve] keys_file and the quota defaults go through.

    stdio is unchanged, byte for byte. It builds no recorder, reads no ContextVar and does not
    load the usage module at all.

  • --rate, --burst and --cost-budget on a key are now enforced over the
    network.
    They were recorded and read by nothing. Measured on a live
    kb serve --transport http --keys-only server: a key created --rate 3/min --burst 4
    answered four calls and then

    HTTP/1.1 429 Too Many Requests
    content-type: application/json
    retry-after: 20
    
    {"jsonrpc":"2.0","id":null,"error":{"code":-32000,"message":"rate limit exceeded for this key: 3/min. retry in 20s"}}
    

    Two buckets per key, filled lazily from two floats each. --rate and --burst bound
    requests; --cost-budget bounds tool TIME, as a duration per period (30s/min), and
    each call is charged how long its body ran. A duration rather than a count because a
    count misprices ask by 8x: it is one request and eight tool bodies, since ask
    reaches its siblings below the wrapper that could have counted them.

    The refusal is at the gate, before the request reaches any tool. So it costs one
    header parse rather than a worker thread, and it covers tools/list and the
    kb://stats resource, which cross no tool wrapper at all. A caller with no valid key
    gets 401 and is never counted against a quota: identity resolves first, which is
    what keeps the bucket map keyed by ids this server minted rather than by anything a
    caller can forge.

    Values are validated now. kb keys create --rate 60 is refused at the flag, naming
    the string, so a typo cannot be minted onto a key that then reads as limited. The same
    parser runs over every stored value when a key file is loaded for serving: a bad value
    exits 1 before the socket binds, and a bad value introduced by a live edit is rejected
    with one warning while the previous keyring keeps serving.

    none on any axis means no limit there, and beats a server default. --burst needs
    --rate: on its own it is the capacity of a bucket that does not exist. The minimum
    burst is 4, because an MCP client spends three requests on the handshake before its
    first tool call.

    Not persisted and not shared between processes: a restart refills every quota, and two
    server processes give each key twice its quota. On the sse transport the 429 message
    is lost and the session closes, which is a defect in that client, not in this server.
    docs/mcp-transports.md carries all three.

    stdio is unchanged, byte for byte. It builds no limiter, opens no timer and does not
    load the rate-limit module at all.

  • [serve] default_rate, default_burst and default_cost_budget in kb.toml, for
    a quota that applies to every key that names none of its own. All three are unset out
    of the box
    , so an upgrade starts limiting nobody. Read only from
    ~/.contextlake/kb.toml or a file passed to --config: a .contextlake.kb.toml found
    by walking up from the current directory is ignored with one line saying so, because a
    rate limit a repository checkout can rewrite is not a limit.

    A shared token is bounded by default_rate and has no per-credential opt-out, since it
    has no key record to write none on.

  • Unknown keys in [serve] are warned about. The table was known but its keys were
    never checked the way [kb] keys are, so default_rat = "60/min" was a silent way to
    leave every key unlimited. It now prints one line naming the key and the known set.

  • --tools and --owners on a key are now enforced over the network. They were
    recorded and read by nothing. Measured on a live kb serve --transport http --keys-only server: a key created --tools none --repos nothing-matches/* used to
    get the full tool list and its calls all ran; the same key on the same server now
    gets an empty tool list and a refusal that names the group which would grant the
    call. --repos and --external still bind nothing and still print
    (recorded, not enforced); --rate, --burst and --cost-budget went live in the
    same release, below.

    Enforced at three surfaces, because a gate on one is a gate the caller walks around
    by using another: the tool wrapper, tools/list, and the kb://stats resource, which
    answers the counts graph_stats answers and crosses no wrapper at all.

    --tools takes comma-separated groups (graph, search, docs, stats, owners,
    semantic) plus all, read and none. read is every group except semantic. A
    group this server does not know is refused at create, so a typo cannot be minted
    onto a key that then reads as scoped. In a hand-edited key file the same value is
    denied rather than refused: it narrows the key and never widens it.

    ask is refused unless every tool it routes to is granted. It calls eight siblings
    directly, below the wrapper that checks a grant, so a key granted ask and denied
    blast_radius would otherwise reach blast_radius through the impact route.

    --owners real allows who_knows; pseudonymous and hidden refuse it, and refuse
    ask with it. There is no anonymiser on the network path, so a key that asked for
    pseudonyms gets no names rather than real ones.

    --repos is deliberately not enforced. It cannot be decided from a call alone: a
    node id does not carry the repository it came from, and repo_dependencies,
    repo_flow and repo_event_flow take a required repo and return rows naming other
    repositories. Correct scoping needs a filter inside the store.

    stdio is unchanged, byte for byte. It reads no key, no policy and no identity, and it
    loads no grant module.

Changed

  • The (recorded, not enforced) label moved from per line to per axis. One label
    after all three scope axes claimed the same thing about all three, so enforcing
    tools alone would have made the line say repos and owners were live too.
    kb keys show now marks each axis on its own, and an unset axis carries no marker at
    all, because it records no scope for a marker to qualify.

  • kb keys show and list print the EFFECTIVE quota and where each value came
    from.
    The limits line used to read a bare unset for a key that named no rate. With
    [serve] default_rate set, such a key is limited, and an operator reading unset
    hands it out believing it is not. It now renders one of four states per axis:
    rate=60/min (enforced), rate=unset -> 60/min from [serve] default_rate (enforced),
    rate=unset (no limit), or rate=none (enforced: no limit, set on the key). The
    rate column in list shows the effective value for the same reason.

    Each per-record --json document gains effective_rate, effective_burst,
    effective_cost_budget (strings or null) and limits_source, an object mapping each
    of the three axes to key, config or unset. No field is removed and none changes
    type.

  • policy_enforced in every --json document is derived rather than a fixed
    false.
    It answers whether every axis the document renders is enforced, so a key
    scoped only on --tools reads `t...

Read more

contextlake 9.1.0

Choose a tag to compare

@github-actions github-actions released this 06 Sep 06:18

Added

  • --json on all seven kb keys verbs. create, revoke, rotate, prune and
    check now emit a document, joining list and show.

    9.0.0 made those five refuse the flag at exit 2. That was the honest interim state: the
    release before it let them take --json, print their ordinary log lines and exit 0, so a
    script that asked for machine-readable output got prose and no error. Refusing was better
    than lying about it, and answering is better than refusing.

    Standard output carries the document and nothing else, on every exit path. A failure is a
    document too, carrying "error" with a snake_case code, which is what kb query,
    kb owners, kb impact and kb eval already do.

    Three fields exist because an exit code could not carry the answer:

    • changed on revoke, rotate, prune and create says whether the key file was
      written. Revoking a key somebody else already revoked exits 0 and changes nothing,
      which read the same to a script as revoking it.
    • reason on check is one of malformed, unknown, revoked or expired. All four
      exit 1, so a CI gate that warns on one and fails on another had nothing to read.
    • last_used_state is not-recorded in this release. last_used_at is null for two
      different reasons and the sibling is the only thing that separates them.

    create --json and rotate --json keep the key on stderr and report
    "key_shown_on": "stderr". --json > out.json would otherwise write a live credential
    into a file at the caller's umask. --print-key moves the key into the document's key
    field, and it already refuses a terminal.

  • rotate honours --print-key and --out. Both flags parsed on rotate and both
    were ignored, exiting 0. The new key exists nowhere else, so a rotation script had no
    route to it but scraping stderr.

Fixed

  • kb keys create --out <existing path> minted a key and lost it. The record was
    written to the key file, and only then was the --out path refused for already existing. The
    command exited 2 saying nothing had worked while a live record sat in the file whose
    plaintext had never been shown to anybody. The output file is now opened before the key
    is minted, and removed again if the mint fails.

    If you ran that command on 9.0.0, look at kb keys list. An orphaned record is
    indistinguishable from a key you hold: same live state, same empty LAST USED, and no
    field on the record says whether the key ever reached anybody. Find the records matching
    the names you tried to create, and kb keys revoke them. Each retry of the failing
    command minted another one, so there may be more than one per name.

  • kb keys show --json on an unknown id emitted no JSON. The not-found branch ran
    ahead of the --json check, so it printed prose to stdout and exited 1. It now emits
    {"error": "unknown_id", ...} at the same exit code, as do revoke and rotate.

  • --overlap help said "default 0". The default is 7d.

contextlake 9.0.0

Choose a tag to compare

@github-actions github-actions released this 06 Sep 03:22

Added

  • contextlake kb keys, the command every key-file refusal already named. Seven verbs:
    create, list, show, revoke, rotate, check and prune.

    kb serve refuses to start on several key-file states, and each refusal told the operator
    to run contextlake kb keys create <name>. That command did not exist. A server that
    refuses and names a way out the reader cannot take is worse than one that starts wrongly,
    because the operator has nothing to do next. This is that command.

    The key is shown once. It goes to standard error at creation and never appears again,
    because the file stores a SHA-256 digest rather than the key. It is deliberately never sent
    through the logger: the console handler always writes to stdout, a --log-file run adds a
    5 MB rotating file with three backups that outlives the process, and the redactor rewrites
    workspace paths and repo names, so it would scrub a key on neither. A lost key is rotated,
    never recovered, and rotate keeps the old key working for --overlap so the holder can
    swap without an outage.

    Two other paths can carry the key out, and both are narrower than they look. --print-key
    writes the bare key to stdout for a pipe and refuses a terminal, where it would land in the
    scrollback instead of a secret store. --out FILE writes it at mode 0600, with the mode set
    at creation rather than chmod-ed afterwards, and with O_EXCL so an existing path is
    refused rather than overwritten.

    check reads the key from standard input only. A key on a command line lands in shell
    history and shows in ps to every account on the machine. A terminal with nothing piped
    in is refused too, rather than waiting for end-of-file behind a blank screen: it does not
    prompt, because a typed key lands in the scrollback. It opens no socket and sends no
    request, which is what lets it answer when the server is the thing that is down, and it
    says so rather than implying it verified anything against a server.

    No verb opens the store database. Every one of them runs on a machine that has never
    built an index, which is the machine an operator is on when a server has just refused to
    start. kb keys list is the first command they run.

    Two refusals split by verb rather than collapsed into one rule. A key file carrying group
    or other bits, or sitting in a directory anyone can write to, is a policy fault: write
    verbs refuse, and list warns and prints the table anyway, because blocking the operator
    from seeing what exists is the wrong failure when list is how they diagnose the refusal
    they just hit. A file that cannot be read at all is not that: nothing was read, so every
    verb fails and names the path.

    The scope flags are recorded and enforced by nothing, and every surface says so.
    --tools, --repos, --owners, --rate, --burst and --cost-budget are written onto
    the key and rendered back by create, list, show and check. No code reads them. A
    key created with --tools none --repos nothing-matches/* was presented to a live
    kb serve --transport http --keys-only server and tools/list answered with all 23
    registered tools, one of which then ran and returned a result.

    So the values print with (recorded, not enforced) beside them and three lines saying
    what that means, show --json and list --json carry "policy_enforced": false, and an
    unset axis reads unset rather than none, which read as a denial for the key a bare
    create makes. Telling an operator their key is scoped when it is not is worse than not
    offering the flags, because they hand the key out on that reading.

    Scope is per tool, not per repository once it does work: a key allowed a tool will read
    every indexed repo through it. --rate and --cost-budget are stored as typed and
    validated by nobody, because their parser ships with the rate limiter. The LAST USED
    column reads never until the usage file it reads from exists.

  • Per-request identity on the network MCP transports, and the frame that will carry access
    control.
    Groundwork only: nothing is enforced yet and nothing changes for a local run.

    build_http_app now resolves a Principal per request and the tool wrapper reads it, so
    two callers holding two different credentials are two different identities inside a tool
    body rather than one anonymous caller. Until now there was no place to put that fact, which
    is why access control, rate limiting and usage accounting could not be built: each would
    have had to invent its own notion of who was asking.

    The wrapper's whole try/finally/except structure lands here, once, deliberately. Four
    planned stories each need to add a line to that twelve-line function, and when they were
    specified separately their orderings contradicted each other. Landing the frame first turns
    each of them into an insertion at a named anchor instead of a restructure, so whoever lands
    second does not have to unpick whoever landed first.

    It fails closed. Whether identity is required is a build-time decision made by
    build_http_app, never inferred from whether an identity happens to be present. A stdio run
    does not read the value at all, and on a network run a missing identity refuses the call
    rather than answering it unscoped. Those two states used to be the same value, which meant a
    server whose identity plumbing broke would have answered every request as if unauthenticated
    access were intended, with every test still passing.

    What it does not do. It detects a MISSING identity. It cannot tell a WRONG one, and on
    the SSE transport the plausible failure is substitution rather than absence. Closing that
    needs a per-connection token compared at the boundary, which is specified and not built.
    ask calls its sibling tools directly, so those legs never cross the wrapper and an access
    check placed there will not cover them; the anchor comment says so. And the run outcome the
    wrapper records has no reader until the usage recorder lands.

    stdio, the default, is unchanged: same tool output, no identity lookup, no new file,
    config field or dependency for anyone who never serves over the network.

  • contextlake kb source wizard. It lists every configured source with a reachability
    mark, then offers to add another and loops until you answer no (pressing enter is no). The
    survey reads the same verify_source path kb doctor and kb source test use, so
    "is this source reachable" has one answer across all three. The add step is kb source add
    run interactively, so the prompts, the literal-secret refusal and the write target are the
    same ones. It needs a terminal: a prompt written to a pipe hangs, so a non-interactive run
    is refused with exit 2 and the flag form to use instead.

Changed

  • kb enrich reports edges to code, not only documents stored. The linking step already
    ran: every enrichment document was matched against the repo's symbol names and the matches
    were stored as documented_by edges. The count was then discarded, so the run could report
    only how many documents came back. A document with no edge to any symbol cannot answer a
    question about the code, and it read as a success.

    The run now prints, per repo, the terms tried, the documents returned and the edges attached
    to code, and closes with a line that puts every targeted repo in one of five buckets:
    enriched, nothing returned, returned but unattached, failed, skipped. The five add up to the
    number of repos the run planned to touch, on every exit path.

    "Returned but unattached" is a state, not a failure. The matcher is whole-word with a
    three-character floor, so a ticket that discusses a repo in prose without naming a symbol
    correctly attaches to nothing. That run still prints .

    A repo whose store or shard write fails is now counted and reported instead of aborting the
    whole run. A run where every repo failed that way exits 1.

    API change: run_enrich_repo returns an EnrichCounts(terms, documents, edges) triple
    instead of the document count alone.

  • kb doctor's per-source line now separates three answers that used to render as one
    .
    A source that was dialled and did not answer keeps . A source of a type with no
    reachability probe (gitlab, zendesk) now draws , matching what kb source test
    already prints for the same case. A source with enabled = false also draws and is not
    dialled at all, matching kb connect and kb ingest, which both skip disabled sources.
    None of the three changes doctor's exit code, which is unchanged and still deliberate.

    What this loses: doctor no longer reports a broken path or an unreachable endpoint on a
    disabled source. Nothing reads that source, so the round trip bought nothing, but the line
    used to be there. Re-enable the source to have it dialled again.

  • The bundled --sample demo fleet moved to a new domain, and two of its repos changed
    shape. Repo ids and symbol names changed, so a script that names one has to be edited.
    The
    old fleet's domain read as a real production estate rather than as an obvious invention, which
    is what demo data has to be. It now models a weather-station monitoring network: stations
    report readings, a forecast service runs a model over them, an ingest pipeline normalises raw
    readings, an alerts service fans out severe-weather notices, and a console UI shows it.

    acme/auth-service   ->  acme/station-registry
    acme/catalog-api    ->  acme/forecast-api    (rebuilt, not renamed; see below)
    acme/payments-api   ->  acme/sensor-ingest   (call edges redirected; see below)
    acme/web-ui         ->  acme/console-ui
    acme/notifications  ->  acme/alerts
    acme/shared-lib     ->  acme/shared-lib      (unchanged)
    demo/app            ->  demo/app             (id unchanged; its two symbols...
    
Read more

contextlake 8.13.0

Choose a tag to compare

@github-actions github-actions released this 01 Sep 20:50

Changed

  • The ambiguity fanout cap is 10, not 6, and stores re-index to pick it up. A call
    reference naming a symbol defined in more than the cap's many places produces no edge at
    all, so the caller is simply absent from "who calls X". Measured with the cap removed on
    the two largest ambiguity contributors in a 717,381-node store, the old cap was dropping
    29.6% and 37.0% of resolvable call references. The comment beside it claimed 21.6%
    and "no knee in the distribution"; both were wrong. There is a knee, and 10 sits on it:
    76.2% and 80.6% of references for 1.22x and 1.52x the ambiguous edges.

    Admitting everything is still wrong, for a sharper reason than the old note gave. The
    uncapped cost is one or two pathological names per repository: 2,864 sites naming a
    symbol with 1,432 definitions produced 4.1M edges by themselves, 92% of that
    repository's uncapped total.

    PARSER_VERSION moves to 12, so existing stores re-index rather than keeping edges
    built under the old cap.

Added

  • Zoom in and zoom out buttons on the graph toolbar. Zoom level could only be chosen
    by wheel or pinch. Pinch is a multipoint gesture, and WCAG 2.5.1 (Level A) wants a
    single-pointer alternative; "Fit to view" sets a zoom level but does not let you pick
    one. Each press steps by a fixed ratio about the viewport centre.

    Wheel zoom got more responsive alongside them: wheelSensitivity was 0.2, so reaching
    a readable scale took a dozen notches and the canvas read as stuck. It is now 1.

Fixed

  • Every graph and dashboard page requested a favicon that did not exist, producing a
    404 in the console on each load. Both now carry an inline SVG icon, which costs no
    request and survives an offline export.

  • kb dashboard --site <dir> overwrote files it did not write. The export writes
    index.html unconditionally, so pointing it at a directory holding anything else --
    a docs site, a hand-maintained landing page -- replaced that content silently, with the
    command reporting success. It now refuses any directory containing files the export does
    not itself produce, naming what it found. Re-running into a previous export stays
    idempotent and needs no flag.

contextlake 8.12.0

Choose a tag to compare

@github-actions github-actions released this 01 Sep 19:34

Added

  • A node in the architecture graph now opens the wiki for its repo, and lands on the
    subsystem page covering that node's file.
    The graph page had no wiki link of any kind.
    Inside the dashboard the graph runs in an iframe, so the node's control asks the
    dashboard to route; the dashboard resolves the file to the narrowest generated
    subsystem page that contains it and opens that, falling back to the repo's own page
    when no subsystem page covers it. Opened through the dashboard's Fullscreen link the
    same page has no parent, so it links that route directly. A static export links the
    sibling wiki page it already writes.

    The control appears only for a repo that has a generated wiki. The prefix match is
    anchored on a path segment, so src cannot claim srcutil/helper.py.

    There is no jump to a heading, and that is a limit of the data rather than an
    omission. Generated pages carry page-level headings only (Overview, Setup & Run,
    Architecture, Dependencies, Gotchas), the model is told to omit any it has nothing to
    say for, and nothing in a page is about a single symbol. A computed anchor would land
    silently at the top of the page.

  • The docs pages that explain the graph now carry the running graph.
    asking-the-graph, code-graph-model, indexing-the-code-graph and
    visualizing-the-graph described the visualizer in prose while the live page sat one
    directory away, reachable only from the landing page. The page is 788 KB, so it is not
    embedded eagerly: the markup ships a screenshot and an IntersectionObserver swaps the
    iframe in when the reader scrolls near it, carrying the current theme. A reader with no
    JavaScript, or who never scrolls that far, keeps the screenshot.

Fixed

  • A static export's graph pages disagreed about which repos had a wiki. The map was
    filled while the pages were being written, so each page saw only the repos written
    before it and the fleet overview, written first, saw none. Nothing read the map yet, so
    nothing failed. It is now built before the first page.

  • RepoTooLarge told you to pass a flag that does not exist. A repository over the
    memory budget was refused with "narrow it with --languages". There has never been a
    --languages flag; the message introduced it, so anyone who followed the advice got
    "unrecognized arguments" and no way to act on the error. The setting is real and lives
    in kb.toml as kb.languages, which the message now names. A guard was added: no
    string inside a raise or an exception's __init__ may name a long flag that no parser
    registers.

  • The command palette's search field removed its own focus ring.
    .cl-palette__input:focus set outline: none, which takes the indicator away from
    keyboard users rather than only from mouse users. It was safe while the palette held one
    focusable control and would have become a WCAG 2.4.7 failure the moment a second one was
    added, silently. Narrowed to :focus-visible, with a stylesheet-wide guard.

contextlake 8.11.0

Choose a tag to compare

@github-actions github-actions released this 31 Aug 16:15

Fixed

  • The dashboard rendered twice on every load, and fetched /api/overview twice with it.
    boot() attaches a hashchange listener and then gives the page a default hash when it
    opens without one. Assigning location.hash fires hashchange, and the listener is
    already attached, so the page rendered once from the explicit call and once from the
    event. The two requests overlap, so neither can serve the other from cache. Measured
    against a 961,633-node store: 2,769 ms and 4,005 ms, 35 ms apart, while the page shell was
    ready in 114 ms. history.replaceState writes the same URL and dispatches nothing. It
    also keeps the default hash out of the history stack, so Back no longer returns to the
    hash-less URL and straight back again.

  • The dashboard reserved layout columns for a sidebar and drawer that were not there.
    The rail becomes position: fixed below 768px and the drawer below 1280px, so both leave
    the grid, but the rules naming their columns applied at every width. Specificity is
    resolved before any media query is considered, so those rules won regardless of source
    order. On a 700px viewport the main content measured 64px wide with the rail collapsed.
    Three of the four rail and drawer combinations were wrong, not the one that was reported.
    All four now use the full width, and 1000px and 1400px were re-checked.

  • kb wiki under-reported how much of a run did not happen. When a repository's
    whole-repo page fails, the run skips that repository's module pages rather than trying
    each one. Those pages were counted nowhere, so the four totals added up to less than the
    run planned and six missing subsystem pages read the same as a repository that had none.
    They are now reported as "N not attempted", kept separate from failures because a page
    nobody tried is a different fact from a page that broke. The all-failed line also called a
    page count "repo(s)", so a run that lost three module pages of one repository announced it
    had failed for three repositories.

Added

  • A truncated graph view now names the node kinds it dropped. "500 of 3,200" says a view
    is partial without saying that the part you came for is the part that is missing: a
    repository with 412 table nodes that renders none of them has an empty ER diagram, and
    the old message could not tell that apart from dropping 412 low-value nodes. The four
    worst losses are logged with the count each kind had, and the full breakdown reaches
    callers as dropped_by_kind. Both sides are counted rather than estimated. The key is
    absent on a complete view, so nothing can render "0 dropped" over a view that dropped
    nothing.

  • A forced-colors block for the dashboard, where there were none. Most of it needed
    nothing: the health chips carry the words Fresh and Stale, the confidence chips carry a
    border style and a clipped glyph, and cards, the rail and the drawer have borders that
    survive a forced palette. The trust bar does not. Its segments are sized by flex and told
    apart by background colour alone, so a forced palette merged them into one bar; they now
    carry a divider, and focus uses the system highlight colour.

Documentation

  • The --repos pattern syntax is documented in full, in
    Mirroring repositories. The page said patterns are
    globs and are anchored; it did not say which wildcards are available. It now lists all
    four (*, ?, [abc], [!abc]) with an example each, and states the four rules that
    govern every pattern: comma-separated, anchored, case-insensitive, and matched against
    both the group-qualified and the local path.

    It also covers the one case the anchoring does not reach. There is no escape character,
    so odd*name matches a repo literally named odd*name and matches oddXname as well,
    and cannot select the first on its own. A one-character set does:
    --repos "odd[*]name". * and ? are legal in a path on Linux and macOS, not on
    Windows.

    Every claim on that page is now pinned by tests/test_repos_pattern_syntax.py.

contextlake 8.10.1

Choose a tag to compare

@github-actions github-actions released this 31 Aug 08:25

Fixed

  • kb index --workspace no longer breaks its worker pool. The cause was
    RepoTooLarge, the exception 8.10.0 added for the memory budget: it could not be
    unpickled. A worker that refuses a repository sends the exception back to the parent,
    and Python rebuilds an exception as cls(*args), where args held only the formatted
    message this class passes to Exception.__init__. The rebuild was three arguments
    short and raised TypeError inside ProcessPoolExecutor's manager thread, which has
    no future to attribute a failure to, so it broke the whole executor: every healthy
    worker was sent SIGTERM and every pending repository failed with A process in the process pool was terminated abruptly. One refused repository ended a 656-repository
    run in about a minute.

    pickle.dumps succeeded on the exception throughout. Only the parent's unpickle
    failed, which is why nothing caught it. The same 17-repository cluster that broke the
    pool twice now completes at --workers 4 with 0 failures, the over-budget repository
    reported as a skip, and all four workers exiting 0.

  • Five more exceptions had the same defect and are fixed with it.
    GrammarNotInstalled is raised on the same indexing path and would break a pool the
    same way when an optional grammar is absent. McpToolError, CircuitOpenError,
    StoreBusy and RunBusy are off that path today. Every exception in the package that
    defines its own __init__ now defines __reduce__, and a test walks the package to
    assert each one survives a round trip, so the next one cannot ship without a sample.

  • A broken worker pool now falls back to serial indexing. That fallback was written
    for this failure and was unreachable: BrokenProcessPool subclasses RuntimeError, so
    the per-repository except Exception caught it, counted one failure against whichever
    repository raised it, and continued, and the handler that re-runs the work-list serially
    never ran once. The failure counters are reset before the serial pass, so a repository
    that genuinely failed before the break is not counted again by it.

  • The reason a pool broke is now reported. str(BrokenProcessPool) is a fixed
    sentence naming no cause, and the real reason is attached as __cause__, which was
    discarded. Reporting only the fixed sentence is why three candidate causes stayed
    unseparated across several investigations.

contextlake 8.10.0

Choose a tag to compare

@github-actions github-actions released this 31 Aug 05:38

Upgrading

This release re-indexes every repository on your next kb index. PARSER_VERSION moves
from 10 to 11 because .config and four sibling extensions now carry settings into the graph,
and no commit moves when an extractor starts reading a file type it previously ignored. Without
the bump an existing store would report every repository "unchanged" and never gain a single
setting.

What that costs, measured on a 660-repository fleet:

  • The rebuild is automatic. kb index reports older parser (10 -> 11) and re-indexes rather
    than skipping; the pass after that is quiet again.
  • The new extensions add roughly 61,000 config_key nodes fleet-wide, dominated by 1,023
    .config files at about 57 settings each.
  • The store-size change was not measured and no figure is given for it here. A full
    re-index of that fleet has not been completed on this machine, and a number nobody measured
    is worse than an absent one.

Known issues

kb index --workspace can break its worker pool at --workers above 1. Reproduced twice
on a 17-repository cluster: a worker dies immediately after a large shard completes, and every
still-pending repository fails with A process in the process pool was terminated abruptly. On
a 656-repository run this failed 640 of them in about a minute.

  • --workers 1 is verified working on the same cluster, including every repository that
    breaks the pool.
  • 2 through 8 are untested. The default is min(8, cpu_count - 1), so a plain
    kb index --workspace is affected.
  • It is not a simple out-of-memory: one reproduction broke with 8.9 GB free, another with
    4.3 GB, and system memory dropped at the instant of the break, which is one worker dying
    rather than the machine starving.
  • The root cause is not established. The candidates not yet separated are a transient
    out-of-memory kill of a single worker, a failure transferring a very large shard back to the
    parent, and a grammar crash.

The max_repo_memory budget added in this release does not prevent it, and the reason is the
limit that budget documents: it is linear in input bytes, while the cost here tracks edge
count. One of the repositories involved estimates 0.86 GB and produces 166,000 edges.

Added

  • .config, .props, .targets, .settings and .plist now reach the XML
    config extractor, and PARSER_VERSION moves to 11 so existing stores get them.

    Without the bump the extraction reaches nobody who already has a store: kb index
    gates re-indexing on the parser stamp, so an already-indexed repository reports
    "unchanged" and never gains a setting. Re-indexing happens automatically on the
    next kb index. Only .xml reached the extractor before, so the canonical .NET
    settings file was contributing nothing: measured across 660 repositories, 1,023 .config files
    produced zero nodes while being exactly the files "where is this setting
    defined" is asked about. .resx is deliberately still excluded, being
    localisation rather than settings and worth roughly 91,000 nodes fleet-wide;
    so are project files, which the manifest extractor owns, and .svg, which is
    XML-shaped graphics.

  • [kb] max_repo_memory, a per-repository memory budget checked before any file
    is parsed.
    max_file_bytes bounds one file and cannot bound a repository that
    is wide rather than deep. The repository that took a 15.4 GB machine down had a
    largest file of 3.57 MB against a 5 MB cap, so that cap never fired once, while
    1,432 XML files averaging 0.42 MB added up to 671 MB. The new budget estimates a
    repository's cost from a stat-only pass, weighting each file kind by measured
    peak memory per byte (code 19.6x, SQL 5.0x, XSD 4.3x, XML 3.5x), and skips the
    repository with its name and the dominant kinds if it would exceed the budget.
    It defaults to 3 GB, taken from the fleet rather than chosen: across 660 real
    repositories the median estimate is near zero and p99 is 1.69 GB, with three
    outliers at 6.09, 6.76 and 7.35 GB. Set it to 0 to disable. The estimate is
    linear while the real cost is not, so it runs low on the largest repositories;
    it is a coarse guard, and the existing shard-item check remains the second layer.

contextlake 8.9.0

Choose a tag to compare

@github-actions github-actions released this 30 Aug 14:39

Added

  • AWS and Azure adapters. --platform aws creates an EventBridge Scheduler
    schedule firing an ECS task; --platform azure creates a Container Apps Job on
    a cron trigger. Both shell out to an already-authenticated aws or az, so
    contextlake still ships no cloud SDK. On EKS and AKS use --platform k8s
    instead: both are Kubernetes, so the CronJob adapter serves them and brings
    concurrencyPolicy: Forbid with it. Neither cloud service has an equivalent of
    Forbid, so two runs there can overlap and the second skips on the store's
    advisory lock rather than never starting, which both adapters report. They round
    differently by design: EventBridge takes rate(N minutes) and rounds to whole
    minutes, while a Container Apps Job trigger is a cron expression and rounds the
    way cron does. Registered but never auto-detected, for the same reason as the
    Kubernetes adapter. Verified by asserting the rendered request documents and the
    exact CLI arguments: there is no account here, so neither is verified by
    execution.

  • A Kubernetes adapter, covering OpenShift as well. Renders a CronJob and
    applies it with kubectl, falling back to oc. One adapter serves both,
    because OpenShift is Kubernetes with a stricter default security context and
    the manifest satisfies the stricter one: no runAsUser, since the restricted
    SCC assigns an arbitrary UID and rejects a pinned one, plus runAsNonRoot,
    a dropped capability set and an fsGroup so the mounted state directory stays
    writable. concurrencyPolicy: Forbid gives single-writer semantics from the
    cluster, so the second of two overlapping runs never starts. State mounts a
    PersistentVolumeClaim rather than an emptyDir, because an ephemeral store
    re-indexes the whole fleet every run. The schedule is a cron expression and
    rounds through the same function the cron adapter uses. Nothing is patched in
    the cluster on its own: changing an interval means re-running
    schedule install, since a background rewrite would need cluster-write rights
    for the life of the schedule. Reachable with --platform k8s and never
    auto-detected: kubectl on a PATH does not mean a schedule belongs in that
    cluster.

  • A Windows adapter, so contextlake schedule works through Task Scheduler.
    Creates a task with schtasks /SC MINUTE /MO n under a \contextlake folder.
    Two limits are reported rather than hidden. /MO counts whole minutes, so an
    interval is rounded the way cron's is, down above a minute and up below it, and
    install says when it rounded. schtasks cannot set StartWhenAvailable, so a
    run missed while the machine was off is lost; the adapter reports that with the
    same phrase cron uses, which is what stops status printing the fact twice. The
    command is quoted with Windows rules rather than POSIX ones, because a venv path
    containing a space is the ordinary case there. Verified by asserting the exact
    schtasks arguments: the development machine is Linux, so this backend is not
    verified by execution.

  • A launchd adapter, so contextlake schedule works on macOS. Renders a
    LaunchAgent plist with StartInterval in seconds, installs it with
    launchctl bootstrap gui/$UID (not the deprecated load, which can return 0
    while doing nothing), and reads the interval back off the installed plist
    rather than reporting what was requested. launchd replays a run missed while
    the machine was asleep, like systemd and unlike cron. schedule status reports
    no next-fire time for it, because launchd exposes none for an interval agent
    and a computed guess would drift from what it actually does. Verified by
    asserting the rendered plist and the exact launchctl arguments: the
    development machine is Linux, so this backend is not verified by execution.

  • schedule list reports units whose job record is gone. state() can only
    answer "is job X installed?", which can only be asked about a job that still
    has a record. The reverse had no reader: delete a record and its unit keeps
    firing on schedule, is absent from list, and uninstall cannot reach it,
    because it resolves a job name through the record that is gone. Adapters gain
    installed_names(), implemented by reading the unit directory for systemd and
    the marked crontab blocks for cron. list names each orphan and its platform,
    and says how to remove it. It also names any platform it could not enumerate:
    skipping a platform and finding nothing on it both produce an empty result, so
    reporting only the empty one would let "never looked" read as "checked, clean". With
    --json the two arrive as _orphaned_units and _unchecked_platforms, added
    alongside the jobs rather than nesting them: a script reading this output keeps
    working. The leading underscore is what makes that safe, since a job name must
    start with an alphanumeric and so can never collide.

  • schedule recommend says when the activity bound was never measured. The
    freshness half of the interval formula needs a count of how many repositories
    changed, which only the index stage records. On an install without the kb
    extra nothing records it, so the bound never engages and the interval rests on
    the duty-cycle floor alone. The activity floor line was omitted entirely in
    that case, which read the same as the bound being switched off. It now states
    that it was not measured and what records it. --json gains an activity
    field reading not-measured, no-change or measured: floor_activity_seconds
    is null for the first two of those, so the number alone could not tell them
    apart.

  • A memory-budget guard on kb index. One repository in a real fleet
    needed more than 9.3 GB in a single worker and never finished; no worker
    count survives a repository that size. A repository whose parsed shard
    exceeds 2,000,000 combined nodes and edges (well above the largest
    repository known to index successfully, at roughly 356,000) is now skipped
    with a named, explicit log line instead of being persisted, and the run
    continues with the rest. This is a guard, not a fix: it stops one
    pathological repository from taking a whole run down, it does not make that
    repository indexable, and the exit code reflects that the run was not fully
    clean.

  • --workers N on kb index and bootstrap. Caps how many repositories the
    index stage parses in parallel. _index_workspace already accepted and honoured
    a workers value, but nothing wired a flag to it, so the default (one fewer than
    the CPU count, capped at 8) could not be overridden anywhere, including on
    bootstrap, the default scheduled job. Lower it to cut peak memory on a large
    fleet or a small machine.

Fixed

  • Every scheduled job read every other job's run history. All jobs append to
    one history file and nothing in a record said which job wrote it. Two things
    followed. decide_kind asks whether a successful full rebuild is older than
    schedule_full_every, so a rebuild run by one job answered that question for
    a job that had never run one and postponed its rebuild by a whole cycle. The
    recommender's median run duration mixed every job's durations, so a
    two-minute kb index and a forty-minute bootstrap produced one interval
    that fitted neither. Records now carry the job name, passed to the child in
    CONTEXTLAKE_SCHEDULE_JOB, and reads are scoped to one job. Records written
    before this carry no job name and count as the default job's, so no existing
    install loses the measurements it has earned. If you created a named job with
    schedule interval on 8.8.0, its earlier records are reattributed to
    default, so that job starts from an empty history and runs one extra full
    rebuild on its next cycle. Every record it writes after that is tagged.

  • kb index's parallel path leaked every completed repository's parsed graph
    for the whole run.
    The worker pool's futs dict was keyed by Future, and
    fut.result() does not clear a future's cached result, so a completed
    repository's GraphShard stayed reachable through futs until the pool's
    with block exited. Measured by A/B on the same 45 repositories at the same
    worker count: 2,130 MB retained versus 95 MB released, 22x apart on
    identical work, growing with repos indexed rather than with worker count.
    Each future's dict entry is now dropped once its (repo_id, path, head) is
    read off it, both on the fast path and the serial fallback after a broken
    pool.

  • A timed-out scheduled run orphaned its worker pool. contextlake schedule run
    spawned its child with subprocess.run(..., timeout=...), which on a timeout kills
    only the direct child. The child is usually bootstrap or kb index, which runs a
    ProcessPoolExecutor of up to 8 workers, so a timed-out run left the whole pool
    running, reparented to init and still holding memory. Measured on a real machine:
    one killed run left 8 orphaned workers holding 12.4 GB. On a schedule, every
    timed-out run leaked another pool. The child now starts in its own process group
    and, on a timeout, the whole group is signalled (SIGTERM, then SIGKILL if it does
    not exit within a few seconds), so nothing below it survives. Windows has neither process
    groups nor SIGKILL, so there the timeout falls back to taskkill /F /T, which
    walks the child tree and reclaims the pool the same way.

contextlake 8.8.0

Choose a tag to compare

@github-actions github-actions released this 27 Aug 17:05

Added

  • contextlake schedule: a self-scheduler. Measures how long a run takes and how often your
    repositories change, works out an interval from the two, and installs a systemd user timer or
    a crontab entry that keeps the mirror and the knowledge layer current on its own, with no unit
    file or cron line to write by hand. Core tier: works without the [kb] extra.
    recommend, status and list read only; install, interval, reset and uninstall
    write. Ad-hoc jobs run any contextlake command on their own interval:

    contextlake schedule interval 6h run -- kb wiki --force

    Ten schedule_* config keys, an honest split between what systemd and cron can each do, and a
    container refusal for state that will not survive a restart. Discarding measured history
    (--purge, reset --history) renames the history file to a .discarded sidecar instead of
    deleting it, so a mistaken discard is recoverable. See Scheduling runs.

  • branch_map: per-repository branch pins. branch puts the whole fleet on one branch,
    which is the common case and not the only one. branch_map takes comma-separated
    pattern=branch pairs using the same globs as repo_filter, so one team can track
    develop while a legacy tree sits on maintenance and everything else follows the
    most-active selection. First match wins, so a specific entry can precede the glob that
    would otherwise catch it. It beats branch, falls back to it, then to the selection.
    A repository whose mapped branch does not exist is reported as unpinned, exactly as
    branch already does, rather than being switched to something else.

    branch_map = team/api=develop, legacy-*=maintenance

Changed

  • bootstrap gained --force. Rebuilds everything instead of only what changed: every
    repository re-parsed, every node re-embedded. It did not exist as a flag before; it is what
    schedule's periodic full cycle runs.

  • configuration.md's example config now shows the scoping keys. repo_filter, branch
    and branch_map were documented in the settings table but absent from the example anyone
    copies, so the two that already existed were easy to miss.