Skip to content

Releases: ShivamPansuriya/jev-skill-gate

v0.2.0 — self-update, stats, and four gating fixes

Choose a tag to compare

@ShivamPansuriya ShivamPansuriya released this 17 Sep 20:32

Install or update without git:

curl -fsSL https://raw.githubusercontent.com/ShivamPansuriya/jev-skill-gate/main/install.sh | bash

Already on v0.2.0 or later: jev-skill-gate update.

New

update — fetches the latest version and migrates on-disk state. No git needed. Safe to run repeatedly: it never downgrades, never removes its own ability to update, rolls back on any failure, is a no-op once current, and takes a lock so two updates cannot interleave. Your stats, config and eval results are never touched.

stats — lifetime tokens saved, spend, and how often gating has run.

  lifetime
    triggered      3 sessions  over 1 day
    tokens saved   30,253  ·  avg 10,084 per session
    manifest       12,750 -> 2,666 avg  (79% smaller)
    spent          $0.0009  ·  $0.03 per 1M tokens saved

$ per 1M tokens saved is the figure that decides whether the API call earns its place.

migrate and config — run state migrations on their own; set provider, base URL, API key and model.

Fixed

Command files were never discovered. Claude Code merged .claude/commands/ into the skill system, so a flat commands/*.md puts its description in context exactly like a SKILL.md. 63 of 217 skills on one machine were ungated.

Four bugs a 54-skill install exposed, reported by a user whose output showed scores of 1.00, 0.98, 0.96 — an arithmetic ramp, not relevance:

  • The safety guard was unfireable for the local scorer. It measured the gap between rank-percentiles, which run 1.00 → 0.00 by construction, so it always looked well separated. It now measures the raw similarities.
  • Caps did not scale down. maxOn: 40 against 54 skills saved 9%. Caps now scale with library size: the same install saves ~78%.
  • The cache ignored the provider, so a keyless run poisoned it for seven days and a later run with an API key silently reused the worse scores.
  • Rank-percentiles were printed as if they were probabilities. Uncalibrated runs now show #1 #2 #3.

Gateway transport was wrong on four counts — endpoint, model location, primitive name and answer field all differ from the TypeSafe direct API. Both transports now work.

Evidence

20 labelled cases, 217 skills, every label validated against the live inventory before the run. Jev scored 4 of them before a free-tier quota ran out: pairwise AUC 1.000, all 19 expected skills survived gating, 0 hidden. The local scorer reached 0.961 across 18 cases.

Verified on live Claude Code sessions: a gated session cannot see the skills the gate hid, and picks the right one when asked.

Reports: RESULTS.md · COMPARISON.md · JEV-PARTIAL.md

43 tests. Node 18+. No dependencies.