Skip to content

kit 5.2.0

Choose a tag to compare

@sandstream sandstream released this 24 Jul 21:50
112cc10

Added

  • kit skill test --runtime now proves negative controls HELD, not just detects violations.
    A forbidden action a gate denied never runs, so it is absent from the transcript — it lands
    in .kit-audit.jsonl as a gate-egress/gate-fs deny. The gate now stamps the PreToolUse
    session_id on that deny event as a join key, and the runtime audit attributes each deny
    to the skill run that was active when it fired (session + span time-window — precise and
    session-bounded, not a global timestamp guess). The negative-control check can now report
    "control held — N forbidden attempt(s) denied" with evidence, closing the path that was
    deferred in the previous increment for lack of a join key. Best-effort: no audit log → no
    denial evidence (never breaks the audit); a denied action is deniedForbidden, never counted
    as a violation that ran. Deterministic, zero-LLM.

  • kit skill test --runtime now checks egress/fs target-scope, not just tool-scope.
    When a signed broker scope is present, each span-attributed action is enriched with a
    broker verdict via the SAME decisions the gate-egress / gate-fs enforcers apply
    (checkEgress/checkFsWrite): a Bash command's hosts, a WebFetch URL, and a
    Write/Edit file_path are checked against the signed [scope]. This catches a skill
    that stays within its declared tools but uses an allowed tool (Bash/WebFetch) to reach
    an off-scope host or write outside the allowed roots — the exfil-via-allowed-tool vector
    — folding it into the existing adherence/negative verdicts. Without a verified scope,
    tool-scope adherence still applies (verdicts stay undefined). Deterministic, zero-LLM.

  • kit skill test --runtime (experimental) — the recorded-run audit that closes two
    of the checks P1 disclaimed.
    kit never runs a skill (zero-LLM); it audits what the
    agent already recorded. The memory transcript index (tool_uses) reconstructs each
    skill run as a span — a Skill invocation opens it, following tool calls until the
    next Skill call or session end are that skill's actions — and two checks now decide
    from that evidence: adherence (an out-of-declared-scope tool that actually ran →
    fail; all in-scope across N observed runs → pass; no recorded run → honest skip)
    and negative controls (a forbidden action that succeeded → fail; none attempted →
    skip "not exercised", never a false-green pass). Low-confidence attribution downgrades
    a would-be fail to an inconclusive skip — kit never blames a skill for an action it
    cannot attribute. Off by default; --runtime opts in. Honest coverage limit surfaced:
    verdicts cover observed runs, not all runs; denial-based "control held" evidence and
    egress/fs target-scope adherence are later increments. rubric stays OUT forever (LLM —
    delegated). Design: kit-research skill-test-p2-runtime-adherence note.


Full changelog: https://github.com/sandstream/kit/blob/v5.2.0/CHANGELOG.md

Verify this release:

git tag -v v5.2.0
npm audit signatures