Generator Phase A: bootstrapping, help, audit, reverse - #302
Merged
Conversation
The generator now generates its own commands — C compiler compiling itself.
New infrastructure:
- HelpFormatter: 374 lines of comprehensive spec docs, example specs,
type reference, workflow guides. Accessible via --help and via
./jtag development/generate/help
- CommandAuditor: scans all 305 commands for conformance (specs,
static accessors, factory functions, any casts). Reports via
--audit and ./jtag development/generate/audit
- Reverse engineer: extracts specs from existing hand-written commands
via --reverse and ./jtag development/generate/reverse
Self-generated commands (specs ate their own dogfood):
- development/generate/help — AI reads spec format, examples, types
- development/generate/audit — AI finds conformance gaps
- development/generate/reverse — AI extracts specs from existing code
- development/generate — existing, now with spec + shared HelpFormatter
Fixes:
- TokenBuilder.defaultValueForType() — complex types get {} or []
instead of undefined (was causing TS compilation errors)
- Auditor recurses nested command dirs (found 305 commands, was 201)
- Consolidated duplicate generateExampleSpec() into HelpFormatter
Contributor
There was a problem hiding this comment.
Pull request overview
This PR expands the command generator toolchain by adding generator-focused development commands (help / audit / reverse-engineer), updates the auto-generated command registries/constants, and refactors the generator CLI to expose richer documentation and auditing workflows.
Changes:
- Add new development commands:
development/generate/help,development/generate/audit, anddevelopment/generate/reverse. - Introduce generator-side helpers (
HelpFormatter,CommandAuditor) and wire them into both CLI generation workflows and server commands. - Update generated registries/schemas/constants to include the new commands.
Reviewed changes
Copilot reviewed 42 out of 42 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| src/shared/generated-command-constants.ts | Adds new command constants for generator help/reverse. |
| src/server/generated.ts | Registers new server commands and updates audit command class mapping. |
| src/browser/generated.ts | Registers new browser commands and updates audit command class mapping. |
| src/generator/TokenBuilder.ts | Refactors default generation logic into a reusable type-based helper. |
| src/generator/HelpFormatter.ts | Adds rich, topic-based help content and template spec generation for the generator CLI/commands. |
| src/generator/CommandGenerator.ts | Refactors CLI entry behavior to support --help, --template, --audit, --reverse. |
| src/generator/CommandAuditor.ts | Adds command/spec auditing + reverse-engineering from existing Types files. |
| src/generator/specs/development-generate.json | Adds generator spec for development/generate (includes templateType/force in spec). |
| src/generator/specs/development-generate-audit.json | Adds generator spec for development/generate/audit. |
| src/generator/specs/development-generate-help.json | Adds generator spec for development/generate/help. |
| src/generator/specs/development-generate-reverse.json | Adds generator spec for development/generate/reverse. |
| src/generated-command-schemas.json | Updates generated command schema registry to include new generator commands and revised audit params. |
| src/commands/development/generate/server/GenerateServerCommand.ts | Uses HelpFormatter templates for --template mode and attempts to support templateType. |
| src/commands/development/generate/audit/shared/DevelopmentGenerateAuditTypes.ts | New shared Types for generator conformance audit command. |
| src/commands/development/generate/audit/server/DevelopmentGenerateAuditServerCommand.ts | Implements command audit via CommandAuditor. |
| src/commands/development/generate/audit/browser/DevelopmentGenerateAuditBrowserCommand.ts | Browser stub delegating audit to server. |
| src/commands/development/generate/audit/README.md | Updates audit command documentation to new behavior/params. |
| src/commands/development/generate/audit/package.json | Updates audit command package metadata/scripts to match new command structure. |
| src/commands/development/generate/audit/.npmignore | Updates ignore rules for the new command package layout. |
| src/commands/development/generate/audit/test/unit/DevelopmentGenerateAuditCommand.test.ts | Adds generated unit test scaffold for audit command. |
| src/commands/development/generate/audit/test/integration/DevelopmentGenerateAuditIntegration.test.ts | Adds generated integration test scaffold for audit command. |
| src/commands/development/generate/audit/test/unit/AuditTypes.test.ts | Removes old vitest-based type tests for prior audit implementation. |
| src/commands/development/generate/audit/test/integration/AuditCommand.test.ts | Removes old vitest-based integration tests for prior audit implementation. |
| src/commands/development/generate/audit/server/GenerateAuditServerCommand.ts | Removes prior audit server implementation (ModuleAuditor-based). |
| src/commands/development/generate/audit/browser/GenerateAuditBrowserCommand.ts | Removes prior audit browser implementation. |
| src/commands/development/generate/audit/shared/GenerateAuditTypes.ts | Removes prior audit Types tied to old implementation/params. |
| src/commands/development/generate/help/shared/DevelopmentGenerateHelpTypes.ts | New shared Types for generator help command. |
| src/commands/development/generate/help/server/DevelopmentGenerateHelpServerCommand.ts | Implements help content retrieval via HelpFormatter. |
| src/commands/development/generate/help/browser/DevelopmentGenerateHelpBrowserCommand.ts | Browser stub delegating help to server. |
| src/commands/development/generate/help/README.md | Adds help command documentation. |
| src/commands/development/generate/help/package.json | Adds help command package metadata/scripts. |
| src/commands/development/generate/help/.npmignore | Adds help command ignore rules. |
| src/commands/development/generate/help/test/unit/DevelopmentGenerateHelpCommand.test.ts | Adds generated unit test scaffold for help command. |
| src/commands/development/generate/help/test/integration/DevelopmentGenerateHelpIntegration.test.ts | Adds generated integration test scaffold for help command. |
| src/commands/development/generate/reverse/shared/DevelopmentGenerateReverseTypes.ts | New shared Types for reverse-engineer command. |
| src/commands/development/generate/reverse/server/DevelopmentGenerateReverseServerCommand.ts | Implements reverse-engineer behavior and optional saving to specs directory. |
| src/commands/development/generate/reverse/browser/DevelopmentGenerateReverseBrowserCommand.ts | Browser stub delegating reverse-engineer to server. |
| src/commands/development/generate/reverse/README.md | Adds reverse-engineer command documentation. |
| src/commands/development/generate/reverse/package.json | Adds reverse-engineer command package metadata/scripts. |
| src/commands/development/generate/reverse/.npmignore | Adds reverse-engineer command ignore rules. |
| src/commands/development/generate/reverse/test/unit/DevelopmentGenerateReverseCommand.test.ts | Adds generated unit test scaffold for reverse-engineer command. |
| src/commands/development/generate/reverse/test/integration/DevelopmentGenerateReverseIntegration.test.ts | Adds generated integration test scaffold for reverse-engineer command. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+27
to
31
| // If template mode, return example CommandSpec from HelpFormatter | ||
| if (params.template) { | ||
| const templateSpec = this.generateExampleSpec(); | ||
| const templateType = ((params as unknown as Record<string, unknown>).templateType as string) || 'standard'; | ||
| const templateSpec = HelpFormatter.templateSpec(templateType); | ||
| return createGenerateResultFromParams(params, { |
There was a problem hiding this comment.
templateType is being read via a cast from params, but it isn’t part of the GenerateParams interface and also isn’t present in the generated command schema for development/generate (generated-command-schemas.json only lists spec and template). As a result, --templateType won’t be accepted/forwarded by normal callers and this will always fall back to 'standard'. Consider adding templateType to GenerateParams (and regenerating schemas) or removing this branch/cast to avoid advertising an option that can’t be set.
Comment on lines
+16
to
+27
| { | ||
| "name": "templateType", | ||
| "type": "'minimal' | 'standard' | 'rust-ipc' | 'browser-only'", | ||
| "optional": true, | ||
| "description": "Type of template to return (default: 'standard')" | ||
| }, | ||
| { | ||
| "name": "force", | ||
| "type": "boolean", | ||
| "optional": true, | ||
| "description": "Overwrite existing command directory if it exists" | ||
| } |
There was a problem hiding this comment.
This spec advertises templateType and force params for development/generate, but the actual command Types/schemas currently expose only spec and template (so callers can’t pass --templateType/--force). Either regenerate/update the development/generate command to match this spec, or trim the spec params to what the command really accepts so the generator docs/examples don’t mislead users.
Comment on lines
+258
to
+267
| static defaultValueForType(type: string): string { | ||
| if (type === 'boolean') return 'false'; | ||
| if (type === 'number') return '0'; | ||
| if (type === 'string') return "''"; | ||
| if (type === 'object') return '{}'; | ||
| if (type.endsWith('[]') || type.startsWith('Array<')) return '[]'; | ||
| if (type.startsWith('Record<')) return '{}'; | ||
| if (type.startsWith("'") || type.includes(" | '")) return "'' as " + type; | ||
| // For complex types, use empty object cast — better than undefined | ||
| return '{} as ' + type; |
There was a problem hiding this comment.
defaultValueForType() returns '' as <union> for string-literal unions. At runtime this produces an empty string that is not a valid member of the union (e.g. 'asc' | 'desc'), which can create invalid payload defaults and hide missing inputs. Prefer choosing a real union member (e.g. first literal) or returning undefined for optional fields rather than casting an invalid value.
Comment on lines
+99
to
+102
| for (const entry of summary.entries) { | ||
| const icon = entry.issues.length === 0 ? ' ' : | ||
| entry.anyCastCount > 0 ? ' ' : ' '; | ||
|
|
There was a problem hiding this comment.
The icon selection currently returns the same string in all branches, so the audit output can’t visually distinguish OK vs warnings vs errors. This looks like a logic mistake (likely meant ✅/issues.length / anyCastCount.
Comment on lines
+282
to
+286
| const hasStaticAccessor = typesContent.includes('.execute(') && typesContent.includes('commandName:'); | ||
|
|
||
| // Check factory functions | ||
| const hasFactoryFunctions = typesContent.includes('createPayload') || typesContent.includes('create') && typesContent.includes('Params'); | ||
|
|
There was a problem hiding this comment.
hasFactoryFunctions is currently inferred via typesContent.includes('createPayload') || (typesContent.includes('create') && typesContent.includes('Params')), which will report “factories present” even when only create*Params (or only create*ResultFromParams) exists. For example, commands/data/clear/shared/DataClearTypes.ts has createDataClearParams but no createDataClearResult; this audit would still treat it as having factory functions. To match the spec (“missing createParams/createResult”), consider explicitly checking for both create<Cmd>Params and create<Cmd>Result (or at least separate checks for params/result factories).
joelteply
added a commit
that referenced
this pull request
Aug 2, 2026
… for the storage tier The gate the NVMe serving-tier eviction (#302) consults before dropping a frozen GGUF: is an IDENTICAL twin already on cold storage? is_structural_twin (pure) = same shard count + per-shard name + size, zero-byte shards never match. scan_shards + find_cold_twin are the thin fs layer. Never drop an NVMe artifact without a VERIFIED cold twin (dropping 662GB on a path guess is the failure this guards). Standalone-validated 5/5. Composes with device_fit + M5's NvmeServingTierPool. Refs #12 #36. Design: STORAGE-SERVING-TIER-GOVERNOR.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q4NU4VNiELPQfBpCacDZGc
joelteply
added a commit
that referenced
this pull request
Aug 2, 2026
…n streaming tier (#302 slice 0) (#2115) BigMama's measurement (2026-08-02): streaming an expert bank off a 130 MB/s HDD = 156-544 s/token = unservable. The prior doc claimed Cold is where 'MoE expert sets are paged into VRAM on demand' — conflating the frozen tier (source GGUFs, backups, unserved models) with the hot per-token-paged tier, which MUST be NVMe. The doc now states the boundary the NvmeServingTierPool (#302) governs: frozen until migrated up. Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
joelteply
added a commit
that referenced
this pull request
Aug 2, 2026
…decided eviction owner (#302, my half) The serving tier's hot set (served GGUFs, expert containers, device-fit overrides) now has a governed eviction owner on the existing ResourcePool contract, closing the 'genome-models' deferred entry in every_cache_class_has_a_decided_eviction_story with exactly the owner it demanded: reference-aware, never blind-delete a served model. - Capacity DERIVED (#287-style): volume total − governed reserve (10% floored at 32 GiB), never a hand-tuned budget. - Relief = MIGRATE coldest-first to the detected COLD drive: copy → fsync → byte-verify → delete source; a verified twin already frozen on cold is a pure drop; verify failure removes the partial COPY, never the source; cold-side name collisions are never clobbered. - ActiveArtifactSet (process singleton, same shape as tracked dirs): serving's ensure_hot_resident (BigMama's half) registers resident paths; eviction skips anything protected, prefix-aware both directions. Poisoned lock fails SAFE (everything protected). - No cold drive ⇒ free 0 LOUDLY — pressure stays visible instead of being 'relieved' by destroying re-fetch-hours artifacts. Composes with device-fit one tier down (Unfittable → grid route), per STORAGE-SERVING-TIER-GOVERNOR.md; Cold stays FROZEN-never-streaming. - Boot wiring beside CargoTargetPool: volume = longest mount-prefix match over detect_drives() (now pub), cold_root = <cold>/continuum-cold/models. 5 new tests in the existing mod (nested serving_tier theme), each pinning a named invariant; the decided-story test STRENGTHENED (owned += genome-models), not weakened. cargo test disk_eviction: 11/11. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
joelteply
added a commit
that referenced
this pull request
Aug 2, 2026
…et (#302 invariant 1, live end-to-end) The reconcile marks the resolved model's GGUF ACTIVE before any spawn touches it (a mid-load migration is the worst case), swaps the registration on model change (old released exactly once), and clears it on both nothing-servable paths. With this, the NvmeServingTierPool's never-migrate-the-served-model invariant is enforced by the LIVE serving path, not just by tests — no dependency on the cross-node half. serving_daemon + disk_eviction suites: 37/37. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
joelteply
added a commit
that referenced
this pull request
Aug 2, 2026
…decided eviction owner (#302, my half) (#2117) * feat(system-resources): NvmeServingTierPool — genome-models gets its decided eviction owner (#302, my half) The serving tier's hot set (served GGUFs, expert containers, device-fit overrides) now has a governed eviction owner on the existing ResourcePool contract, closing the 'genome-models' deferred entry in every_cache_class_has_a_decided_eviction_story with exactly the owner it demanded: reference-aware, never blind-delete a served model. - Capacity DERIVED (#287-style): volume total − governed reserve (10% floored at 32 GiB), never a hand-tuned budget. - Relief = MIGRATE coldest-first to the detected COLD drive: copy → fsync → byte-verify → delete source; a verified twin already frozen on cold is a pure drop; verify failure removes the partial COPY, never the source; cold-side name collisions are never clobbered. - ActiveArtifactSet (process singleton, same shape as tracked dirs): serving's ensure_hot_resident (BigMama's half) registers resident paths; eviction skips anything protected, prefix-aware both directions. Poisoned lock fails SAFE (everything protected). - No cold drive ⇒ free 0 LOUDLY — pressure stays visible instead of being 'relieved' by destroying re-fetch-hours artifacts. Composes with device-fit one tier down (Unfittable → grid route), per STORAGE-SERVING-TIER-GOVERNOR.md; Cold stays FROZEN-never-streaming. - Boot wiring beside CargoTargetPool: volume = longest mount-prefix match over detect_drives() (now pub), cold_root = <cold>/continuum-cold/models. 5 new tests in the existing mod (nested serving_tier theme), each pinning a named invariant; the decided-story test STRENGTHENED (owned += genome-models), not weakened. cargo test disk_eviction: 11/11. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo * feat(serving): register the resident model's artifact in the active set (#302 invariant 1, live end-to-end) The reconcile marks the resolved model's GGUF ACTIVE before any spawn touches it (a mid-load migration is the worst case), swaps the registration on model change (old released exactly once), and clears it on both nothing-servable paths. With this, the NvmeServingTierPool's never-migrate-the-served-model invariant is enforced by the LIVE serving path, not just by tests — no dependency on the cross-node half. serving_daemon + disk_eviction suites: 37/37. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
joelteply
added a commit
that referenced
this pull request
Aug 3, 2026
#2109) * chore(k3): bump llama.cpp submodule to eee635ba2 — K3 serving stack onto canary Advances the vendored llama.cpp fork 30 commits (clean FF over canary's stale 66594cc3f): container-serve resident-override (LLAMA_RESIDENT_OVERRIDE), the rung-2 ResidencyCache plan-file consumer, the score-hint/generation-bias actuator, PagerCaptureEvent emit, fit-device --reserve-gb. Makes canary USE the K3 misfit-serving stack (measured 0.33 tok/s WASTE-parity on a 32GB card). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q4NU4VNiELPQfBpCacDZGc * docs(pager): RUN-1 K3 trace fixture + tkey->(layer,matrix) table for M5's replay Live GGML_MOE_TRACE_FILE slice (12B records: u64 tkey + u32 e) + the reverse table so BanditPlanController recovers (layer,expert): tkey=FNV-1a of blk.{layer}.ffn_{gate,up,down}_exps.weight, e=within-layer expert idx, expert identity=(layer,e) deduped across the 3 matrices. RUN-1 static-pin datum input. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q4NU4VNiELPQfBpCacDZGc * docs(pager): reference RL-policy prototypes for M5's TierPolicy port The actual std-only Rust prototypes written against live K3 traces this session: trace_replay (recency beats LFU 3-4x), predictor (offline learned-decay +5pts held-out), online_predictor (bandit 49.8 vs 47.8 best-fixed on non-stationary), self_optimize (joint speed×quality). These are the faithful-port source for the learned policy behind TierPolicy (continuum-core expert_tier_policy.rs, #276). Numbers are properties of these exact constants + reward math — reproduce before improving. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q4NU4VNiELPQfBpCacDZGc * docs(k3): GPU-resident hot experts design (task #23 'trend to full GPU') The major GPU speedup: promote hot experts to persistent VRAM so decode's hot path is GPU-native (zero fetch, zero copy). 3 increments (copy-skip -> VRAM hot cache -> pipeline), the 32GB rate-distortion constraint (imatrix-enabled resident shrink frees VRAM for the hot set), measured per-increment via k3-bench. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q4NU4VNiELPQfBpCacDZGc * docs(k3): flag the input_cpy-persistence question gating increment 1 vs 2 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q4NU4VNiELPQfBpCacDZGc * docs(k3): modular rework-proof impl for #23 — reuse ResidencyCache + DeviceUploadFetcher Mechanism is the existing (buft,fetcher)-generic ResidencyCache; a VRAM cache = same class + device buft + host->device fetcher. 3 small parameterized pieces (DeviceUploadFetcher, instantiate w/ GGML_MOE_VRAM_CACHE_GB, seam hook). Stats only tune params -> zero mechanism rework. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q4NU4VNiELPQfBpCacDZGc * docs(arch): MoE serving on a governed budget (draft; M5 owns the governor seam) Diagnoses the hardcoded-cache overcommit that collapsed K3 fetch bandwidth (40GB pinned + mmap = 95.9GB on 63GB -> pagefile thrash -> 205 MB/s -> 0.027 tok/s) and lays out the clean architecture: governor owns the residency budget net of the model's mmap footprint, plan-file is the one wire, ResidencyCache is pure mechanism. Governor-interface sections marked [M5 OWNS] for her to edit. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q4NU4VNiELPQfBpCacDZGc * docs(arch): answer the [M5 OWNS] governor-budget seam in place (net-of-mmap is explicit arithmetic; plan_file.budget_bytes is the lease wire) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo * docs(arch): measured governed-budget inputs + graduated serving/load path Records the BigMama measurements feeding M5's #287 derivation (non-cache ~56GB, per-token working set 5.5GB, governed budget ~6GB, fetch recovers to 2.5GB/s at fit), the now-complete C++ cache mechanism (enable-from-plan, grow, shrink), and the three-piece graduated path to serving/load kimi-k3 (catalog row + serving-lane MoE launch + #287) replacing the rigged .bat. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q4NU4VNiELPQfBpCacDZGc * windows: make continuum-core build + link on windows-msvc (first time) start-server.sh now provisions the full Windows CUDA build env before the cargo builds (the cargo/nvcc path had none, unlike the vcvars-wrapped llama cmake): import MSVC via vswhere->VS2022-14.4x + a .bat env dump (cl.exe for nvcc), pin CMAKE to the manifest install, force CMAKE_GENERATOR=Ninja (the VS18-2026 auto- pick is undefined in cmake 3.30), add the Windows SDK bin (mt.exe/rc.exe), select a complete CUDA toolkit + CUDA_PATH (a provisioning split left cuda-env with 0 import libs vs cuda-13.2's 12), and RUSTFLAGS -L for pocket-tts (which emits no link-search) while re-carrying +crt-static so the /MT GPU stack still links. Portability: expert_container.rs + commands/capacity.rs used Unix-only std::os::unix::fs::FileExt::read_exact_at. Add crate::platform_io::pread_exact (unix read_exact_at / windows seek_read loop) - one place for positioned reads. Build validated (npm start exit 0, continuum-core lib clean). A separate runtime hot-loop on the #2088 core at startup is tracked apart from this build fix. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q4NU4VNiELPQfBpCacDZGc * feat(capacity): device_fit VRAM-partition calc for the governor Pure calc the governor uses to fit a streaming-MoE's RESIDENT (non-expert) tier to a device VRAM budget and reconcile it with the expert tier on ONE budget — fixing the double-count where the expert pager was handed the full VRAM ceiling while resident silently ate most of it. Partition (in order): compute reserve -> resident (Native | device-fit Override | Unfittable) -> sufficient-context KV -> everything left = hot-expert VRAM budget (maximized: more on-GPU experts, fewer streams). Context is derived + clamped, never hand-picked. Artifact resolver injected (no hardcoded paths). Standalone-validated 7/7; M5 wires it into the daemon spawn path + launch (ServingTarget.resident_override) per the K3 sprint split. Refs #29 #31 #36. Arch-confirmed on real K3 UD-IQ2 (93 blk/896 exp/top-16). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q4NU4VNiELPQfBpCacDZGc * feat(serving): resident-override plumbing on ServingTarget + launcher Wire foundation for the governor's device_fit plan: ServingTarget carries resident_override: Option<PathBuf>, and the launcher exports it as LLAMA_RESIDENT_OVERRIDE so llama.cpp sources the precision-shrunk RESIDENT (non-expert) tensors from the device-fit GGUF (all offloaded to GPU) while the primary streams experts. All builders updated; defaults None (resident serves as-shipped, no behavior change) until compute_resident_override + the resolve-or-generate resolver (#35) land next. In-crate validated. Refs #29 #36. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q4NU4VNiELPQfBpCacDZGc * chore(vendor): bump llama.cpp fork to k3-adopt e3ce51df5 M5's per-layer KV accessors (n_head_kv_il + n_embd_head_{k,v}_il, continuum #238) + graph reconciliation. The K3 engine now builds against these — enables the device_fit resident-override serve + honest per-layer K3 KV sizing. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q4NU4VNiELPQfBpCacDZGc * feat(serving): compute_resident_override — wire device_fit into the plan The governor now DECIDES the resident source per serve: compute_resident_override derives resident_bytes (weights - expert_bytes_total) vs the governed VRAM ceiling via capacity::device_fit, and sets ServingTarget.resident_override. A dense/small model fits native (None); a >VRAM-resident MoE (K3) resolves a cached device-fit override that fits, else Unfittable → route to grid / generate (#35), glass-boxed. resolve_device_fit_override (model_registry::artifacts): looks up a per-user device-fit cache convention (<storage_root>/device-fit/<id>/) + a resident-bytes sidecar; returns the override only when its resident fits the usable budget. No hardcoded paths; generation/HF discovery is #35. The resident-fit decision turns only on resident_bytes vs budget — per-layer KV (#2107 ModelCapabilities) drives the context/expert split elsewhere, so KV is not consulted here. Refs #29 #35 #36. In-crate validated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q4NU4VNiELPQfBpCacDZGc * docs(arch): storage serving-tier governor — NVMe<->cold contention managed like VRAM/RAM Joel: 'like vram and memory, this contention has to be managed between cold storage and nvme.' Design: NVMe is a governed HOT-SERVING tier (a ResourcePool, same TrackedDir + evict_at_least machinery as CargoTargetPool), whose eviction = MIGRATE frozen/duplicate artifacts to the Cold drive, not a manual rm. Serving asks ensure_hot_resident(model); composes with device_fit's Unfittable one tier down (VRAM). Corrects the DriveRole bug: Cold (HDD) is FROZEN storage, never the per-token streaming tier (HDD = unservable). Dissolves today's K3 container disk fight: the C: IQ2 is a verified duplicate of the D: copy -> governor migrates it off NVMe -> container fits, no human deletes anything. Refs #12 #36. Design for M5's system_resources lane. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q4NU4VNiELPQfBpCacDZGc * feat(capacity): verified cold-twin detection — safe-to-drop primitive for the storage tier The gate the NVMe serving-tier eviction (#302) consults before dropping a frozen GGUF: is an IDENTICAL twin already on cold storage? is_structural_twin (pure) = same shard count + per-shard name + size, zero-byte shards never match. scan_shards + find_cold_twin are the thin fs layer. Never drop an NVMe artifact without a VERIFIED cold twin (dropping 662GB on a path guess is the failure this guards). Standalone-validated 5/5. Composes with device_fit + M5's NvmeServingTierPool. Refs #12 #36. Design: STORAGE-SERVING-TIER-GOVERNOR.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q4NU4VNiELPQfBpCacDZGc * chore(vendor): bump llama.cpp fork to k3-adopt c6469d5 — container-serve wired Both halves of the DirContainerFetcher wire (BigMama fetcher + moe_pick_fetcher branch 175ac9d6a; M5 caller-side encode + record_bytes reader c6469d5). Serving now reads the aligned per-layer container (GGML_MOE_CONTAINER) instead of the scattered raw GGUF — the honest ~2.6GB/s path. Retires the built-not-wired ContainerFetcher. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q4NU4VNiELPQfBpCacDZGc * fix(serving): resident_override on vision_sidecar ServingTarget + K3 coverage measurement Merge fix: vision_sidecar's ServingTarget was missing resident_override (added by #29). Plus a measurement test that drains the real K3 routed-access fixture through the #282 predictive instrument and prints repeat_recall / predicted_delta / schedulable_coverage — the go/no-go for the LiveUploadPager predictive pipeline (H2D/token = (1 - coverage) x ~11GB). Prints, never asserts (real routing sample). NOTE: can't run on windows-msvc (pre-existing cargo-test Unix-socket block, ipc/mod.rs); runs on M5's Mac. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q4NU4VNiELPQfBpCacDZGc * feat(pager-driver): offline warm-coverage measurement (--synth-layers, --once, --budget-slots) The moe-pager-driver gains an offline replay mode so any completed GGML_MOE_TRACE_FILE can be scored on any box (windows-msvc clean by crate constraint), not just tailed live next to a serve: - --synth-layers N: synthesize the tkey->layer map from layer count alone (TkeyTable::for_layers, the same zero-config seam MoeTraceTail owns) instead of requiring an operator tkey-to-layer-matrix.json. - --once: exit when the trace stops growing (EOF) and print a SUMMARY line with mean DECODE-token serving hit = warm schedulable coverage. - --budget-slots N: override the predictor residency budget (default auto = first token x1.5) to measure the coverage-vs-free-VRAM curve (the device-fit tradeoff). Measured on BigMama run2.trace (302 warm decode tokens): bandit coverage 13.8% @250 slots -> 51.3% @2000 -> 65.7% @4024, beating naive last-N recency by +7-9pts at matched VRAM. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q4NU4VNiELPQfBpCacDZGc * feat(pager): measure cross-layer prefetch predictor ceiling — DEAD lever for K3 VDD offline measurement (cooccur-ceiling bin) on the real warm serve trace (run2.trace, 122 held-out decode tokens, 11102 layer-steps): cross_layer_cooccur_hit 0.159 (adjacent-layer noisy-OR) recency_same_layer_hit 0.403 (last token, same layer) structure beyond recency -0.244 cooccur_recall_on_recency_misses 0.112 (11824/106058) Adjacent-layer co-occurrence predicts <half what plain recency does, and recovers only 11% of the experts recency misses (~base rate). K3 expert routing has no exploitable cross-layer structure — the CrossLayerExpert- Predictor prefetch lever is not worth wiring (saves the ggml pass-id capture slice). Recency-family residency (the bandit EMA curve) is THE signal; the only lever that lifts K3 is freeing VRAM (device-fit shrink) so residency coverage can reach the measured 51%. Caveat: adjacent-layer, one workload trace. Wider-predecessor noisy-OR would regress toward the frequency baseline (which underperforms recency), so a large lift is unlikely — but not measured here. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q4NU4VNiELPQfBpCacDZGc * chore(vendor): bump llama.cpp to de29843e0 — device-resident expert cache half (#23) Pins the fork at the DeviceUploadFetcher wiring (my half of the LiveUpload- Pager H2D-kill). Off unless GGML_MOE_VRAM_CACHE_GB / plan device_budget_bytes enables it; host serving path byte-for-byte unchanged. M5's expert-loop D2D half lands next on the same seam. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q4NU4VNiELPQfBpCacDZGc * chore(vendor): bump llama.cpp to 0fbe4e27a — quantize --resident-only + tier manifest (#40) Enables the device-fit division: produce a small resident override per precision tier + a (tier_label, resident_bytes) sidecar the governor reads to co-optimize the VRAM split. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q4NU4VNiELPQfBpCacDZGc * feat(pager): DivisionPolicy — the governor's VRAM-division RL brain (#2/#3) The second control rung above the pager's DecayBandit. The pager decides WHICH experts stay resident (reward=hit-rate, cheap, online). This decides HOW TO DIVIDE the card — resident (non-expert) weights vs expert cache — to MAXIMIZE tok/s. That reward (actual tok/s) is EXPENSIVE (a serve), so naive online RL flails; the fix is SIM-WARM-START: predict tok/s per division OFFLINE from the measured coverage curve, then a slow bandit refines each arm from real measured tok/s. - CoverageModel: piecewise-linear coverage(slots) over MEASURED points (k3_measured() = the trace-replay curve); saturates, never extrapolates up. - predict_tok_s: coverage -> (1-coverage)*experts/token*expert_bytes H2D -> t_token -> tok/s. Higher coverage -> less H2D -> faster (the load-bearing property, tested). - feasible_divisions: tier catalog (from --resident-only manifests) x HardwareBudget -> cache budget/slots per tier; drops VRAM-overflow tiers. - DivisionBandit: warm_start from the predictor; observe(tier, measured_tok_s) overrides the prior on first serve then EMAs — the expensive reward spent only on the arm actually run. Policy lives here (windows-clean, 4 tests pass); serving_daemon actuates it (M5's #2: discover manifests, feed catalog+budget+live tok/s, apply the chosen {resident_tier, device_budget_bytes} to the plan). Fractal control law: pager (experts<->hit-rate) -> this (VRAM split<->tok/s) -> grid. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q4NU4VNiELPQfBpCacDZGc * chore(vendor): bump llama.cpp to 2a32025dd — device-cache un-crashable (clamp to free VRAM + null-buffer guard, #23) Testing convicted the segfault as VRAM oversubscription (K3 33GB resident + env cache on a 32GB card, cudaMalloc lazy-VMM deferred fault). Fix: clamp device budget to measured free VRAM (mine) + M5's D2D null-buffer guard. Device cache now disables safely where there's no room (K3) and works where there is (V4-Flash); can't crash from any budget source. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q4NU4VNiELPQfBpCacDZGc * test(division): bandit learns residency saturation from measured V4-Flash curve Feeds the real BigMama RTX 5090 --n-cpu-moe sweep (DeepSeek-V4-Flash UD-IQ2_M) into DivisionBandit: 0 resident=1.39, 8 resident=1.69, 14 resident=1.68 tok/s. Asserts the bandit converges on the SATURATION KNEE (8 layers), not max residency — 8->14 layers buys nothing at +11GB VRAM. Encodes the measured finding that the governor must learn 'minimal static residency + max device cache', the freed VRAM belonging to the recency cache (#43), not to over-pinned static layers. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q4NU4VNiELPQfBpCacDZGc * test(division): bandit finds non-monotonic device-cache budget optimum Measured V4-Flash device-cache coverage curve (5090, GGML_MOE_VRAM_CACHE_GB sweep): 6GB/992slots=1.80, 12GB/1985=3.10, 22GB/3630=2.96 tok/s, all 100% hit. tok/s is NON-MONOTONIC in budget: undersized churns, 12GB is the plateau knee, 22GB is no better (100% hit but O(slots) reserve_slot eviction scan). predict_tok_s's monotonic prior would pick 22GB; only the measured reward lands on 12GB — which frees ~20GB of a 32GB card for co-resident lanes. Pins the invariant that the governor must not oversize the cache and starve other models. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q4NU4VNiELPQfBpCacDZGc * chore(vendor): bump llama.cpp to fa7e0d8e9 — #43 device-cache fix + async restore + enum fix Includes the prefetch host_visible guard (THE #43 crash fix, validated 3.05 tok/s V4-Flash device cache on the 5090), M5's async cpy_tensor_async restore, and the moe-pack quant-enum fix. A fresh parent build now includes the un-crashable device cache instead of the pre-fix pin. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q4NU4VNiELPQfBpCacDZGc --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
undefined)New commands (self-generated)
./jtag development/generate/help— AI-readable spec documentation./jtag development/generate/audit— conformance report./jtag development/generate/reverse— extract spec from existing codeAudit baseline
Test plan
npx tsx generator/CommandGenerator.ts --help— full docs rendernpx tsx generator/CommandGenerator.ts --audit— scans 305 commandsnpx tsx generator/CommandGenerator.ts --reverse commands/ping— extracts specnpm run build:ts— clean compilationnpm start+./jtag development/generate/help --topic=spec— live test./jtag development/generate/audit --format=summary— live test./jtag development/generate/reverse --commandDir=commands/ping— live test