Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
100 commits
Select commit Hold shift + click to select a range
81dbaa3
chore(porch): 1286 init aspir
waleedkadous Jul 29, 2026
e5f2d69
[Spec 1286] Initial specification draft
waleedkadous Jul 29, 2026
afadea2
chore(porch): 1286 specify build-complete
waleedkadous Jul 29, 2026
5a461a6
[Spec 1286] Specification with multi-agent review
waleedkadous Jul 29, 2026
084e6ac
chore(porch): 1286 specify re-iter (iter 2)
waleedkadous Jul 29, 2026
3352b3e
chore(porch): 1286 specify build-complete
waleedkadous Jul 29, 2026
aacd878
[Spec 1286] Specification with iteration 2 review fixes
waleedkadous Jul 29, 2026
53f55ad
chore(porch): 1286 specify re-iter (iter 3)
waleedkadous Jul 29, 2026
24b9e36
chore(porch): 1286 specify build-complete
waleedkadous Jul 29, 2026
c2256bf
[Spec 1286] Specification with iteration 3 review fixes
waleedkadous Jul 29, 2026
4bb29b3
chore(porch): 1286 specify force-advance (safety ceiling reached at i…
waleedkadous Jul 29, 2026
c802c8e
chore(porch): 1286 plan phase-transition
waleedkadous Jul 29, 2026
bb9e2c9
[Spec 1286] Thread: specify phase outcome and force-advance note
waleedkadous Jul 29, 2026
a388640
[Spec 1286] Architect-requested codex recheck: trim migration mechani…
waleedkadous Jul 29, 2026
2ae3fc9
[Spec 1286] Thread: codex recheck outcome
waleedkadous Jul 29, 2026
fc86bc0
chore(porch): 1286 plan build-complete
waleedkadous Jul 29, 2026
3fe8d87
[Spec 1286] Thread: note incoming #1288 defaults change
waleedkadous Jul 29, 2026
bf47000
[Spec 1286] Plan with multi-agent review
waleedkadous Jul 29, 2026
8ed853c
chore(porch): 1286 plan re-iter (iter 2)
waleedkadous Jul 29, 2026
076793c
chore(porch): 1286 plan build-complete
waleedkadous Jul 29, 2026
7be20f0
[Spec 1286] Plan with iteration 2 review fixes
waleedkadous Jul 29, 2026
6f61c9a
chore(porch): 1286 implement phase-transition
waleedkadous Jul 29, 2026
63e15f7
[Spec 1286][Phase: phase_1] feat: config schema, validators, and lane…
waleedkadous Jul 29, 2026
1f9684d
chore(porch): 1286 implement build-complete
waleedkadous Jul 29, 2026
bf57b66
[Spec 1286] Thread: phase_1 and the baseline-landmine ruling
waleedkadous Jul 29, 2026
a49ba29
[Spec 1286][Phase: phase_1] fix: null guard on nested modelsByType; f…
waleedkadous Jul 29, 2026
726f6c9
chore(porch): 1286 implement re-iter (iter 2)
waleedkadous Jul 29, 2026
3dee228
[Spec 1286] Thread: phase_1 review outcome and hold on #1290
waleedkadous Jul 29, 2026
6e91432
[Spec 1286] Thread: #1290 now green, blocked only on review
waleedkadous Jul 29, 2026
c834151
chore(porch): 1286 implement build-complete
waleedkadous Jul 30, 2026
2fbcf50
[Spec 1286] Thread: rebase onto main, phase_1 checks green
waleedkadous Jul 30, 2026
5fdcd6f
[Spec 1286][Phase: phase_1] fix: reject empty lane lists; drop dead n…
waleedkadous Jul 30, 2026
d7bad51
chore(porch): 1286 advance plan phase → phase_2
waleedkadous Jul 30, 2026
c85ab49
[Spec 1286] Thread: phase_1 approved, empty-list disclosure
waleedkadous Jul 30, 2026
a0bbdce
[Spec 1286][Phase: phase_1] docs+test: pin the config-vs-protocol asy…
waleedkadous Jul 30, 2026
3a49f76
[Spec 1286] Thread: the [] asymmetry ruling and its premise guard
waleedkadous Jul 30, 2026
eb4a512
[Spec 1286][Phase: phase_2] feat: per-lane model wiring for the claud…
waleedkadous Jul 30, 2026
ea945af
[Spec 1286] Thread: phase_2 and the silently-dropped flag
waleedkadous Jul 30, 2026
580f068
chore(porch): 1286 implement build-complete
waleedkadous Jul 30, 2026
2c27492
[Spec 1286][Phase: phase_2] refactor: extract consult flag mapping so…
waleedkadous Jul 30, 2026
ca51d90
[Spec 1286] Thread: extraction ruling and the second build-vs-test race
waleedkadous Jul 30, 2026
b29b40e
[Spec 1286][Phase: phase_2] fix: refuse --model-id for lanes with no …
waleedkadous Jul 30, 2026
482e2f7
[Spec 1286] Thread: phase_2 iter1 verdicts and the hermes inert-flag fix
waleedkadous Jul 30, 2026
0543072
[Spec 1286][Phase: phase_2] docs: iter1 rebuttal — codex hermes findi…
waleedkadous Jul 30, 2026
0eb0a03
chore(porch): 1286 implement re-iter (iter 2)
waleedkadous Jul 30, 2026
9fe6667
chore(porch): 1286 implement build-complete
waleedkadous Jul 30, 2026
922114e
chore(porch): 1286 advance plan phase → phase_3
waleedkadous Jul 30, 2026
bc5772a
[Spec 1286][Phase: phase_3] feat: agy lane model passthrough and the …
waleedkadous Aug 1, 2026
691e708
[Spec 1286] Thread: phase_3 split and the stderr diagnostic
waleedkadous Aug 1, 2026
a3e975e
chore(porch): 1286 implement build-complete
waleedkadous Aug 1, 2026
d266e65
[Spec 1286][Phase: phase_3] fix: four review findings on the fail-fas…
waleedkadous Aug 1, 2026
55d6212
[Spec 1286][Phase: phase_3] docs: iter1 rebuttal — four findings acce…
waleedkadous Aug 1, 2026
514b066
chore(porch): 1286 implement re-iter (iter 2)
waleedkadous Aug 1, 2026
a84f797
chore(porch): 1286 implement build-complete
waleedkadous Aug 1, 2026
22cd65f
[Spec 1286][Phase: phase_3] fix: claude's three minor notes on the split
waleedkadous Aug 1, 2026
e5f017d
[Spec 1286] Thread: phase_3 approved and the invariant-vs-comment lesson
waleedkadous Aug 1, 2026
a1c6940
chore(porch): 1286 advance plan phase → phase_4
waleedkadous Aug 1, 2026
cfa2089
[Spec 1286][Phase: phase_4] feat: model_id metrics column and honest …
waleedkadous Aug 1, 2026
062a547
chore(porch): 1286 implement build-complete
waleedkadous Aug 1, 2026
b3f4950
Merge remote-tracking branch 'origin/main' into builder/aspir-1286
waleedkadous Aug 1, 2026
b835748
[Spec 1286][Phase: phase_4] fix: migration race and four review findings
waleedkadous Aug 1, 2026
7933207
Merge remote-tracking branch 'origin/main' into builder/aspir-1286
waleedkadous Aug 3, 2026
aa7a52d
[Spec 1286][Phase: phase_4] docs: thread notes on the upstream test-t…
waleedkadous Aug 3, 2026
19ab97f
[Spec 1286][Phase: phase_4] docs: iter1 rebuttals — all findings acce…
waleedkadous Aug 3, 2026
b5c9d11
chore(porch): 1286 implement re-iter (iter 2)
waleedkadous Aug 3, 2026
af49d76
chore(porch): 1286 implement build-complete
waleedkadous Aug 3, 2026
bc7a1c2
[Spec 1286][Phase: phase_4] fix: run concurrency children against sou…
waleedkadous Aug 3, 2026
a7710d0
[Spec 1286][Phase: phase_4] docs: split iter2 rebuttals into their ow…
waleedkadous Aug 3, 2026
0b5410f
chore(porch): 1286 implement re-iter (iter 3)
waleedkadous Aug 3, 2026
9da0c08
chore(porch): 1286 implement build-complete
waleedkadous Aug 3, 2026
b17d91c
[Spec 1286][Phase: phase_4] test: reap barrier orphans, yield instead…
waleedkadous Aug 3, 2026
9125b26
chore(porch): 1286 advance plan phase → phase_5
waleedkadous Aug 3, 2026
af4cd65
[Spec 1286][Phase: phase_5] refactor: one lane-selection resolver for…
waleedkadous Aug 3, 2026
c8c7cd7
chore(porch): 1286 implement build-complete
waleedkadous Aug 3, 2026
513a028
[Spec 1286][Phase: phase_5] docs: thread notes on resolver consolidation
waleedkadous Aug 3, 2026
8649bf4
[Spec 1286][Phase: phase_5] fix: mock process.exit in done tests; rep…
waleedkadous Aug 3, 2026
0d1200e
chore(porch): 1286 implement re-iter (iter 2)
waleedkadous Aug 3, 2026
2f4a68f
chore(porch): 1286 implement build-complete
waleedkadous Aug 3, 2026
212a6bb
[Spec 1286][Phase: phase_5] fix: reuse verifyConfig; correct an overs…
waleedkadous Aug 3, 2026
ebf94c0
[Spec 1286][Phase: phase_5] docs: thread notes on phase_5 approval
waleedkadous Aug 3, 2026
0ea08e1
chore(porch): 1286 advance plan phase → phase_6
waleedkadous Aug 3, 2026
66e834d
[Spec 1286][Phase: phase_6] docs: config reference, precedence ladder…
waleedkadous Aug 3, 2026
d5d0fcb
chore(porch): 1286 implement build-complete
waleedkadous Aug 3, 2026
056f579
[Spec 1286][Phase: phase_6] docs: fix config layers, document --model…
waleedkadous Aug 3, 2026
48bdc32
chore(porch): 1286 implement re-iter (iter 2)
waleedkadous Aug 3, 2026
916cc85
chore(porch): 1286 implement build-complete
waleedkadous Aug 3, 2026
d554f99
[Spec 1286][Phase: phase_6] docs: fix a non-running CLI example; comp…
waleedkadous Aug 3, 2026
1365c9d
chore(porch): 1286 implement re-iter (iter 3)
waleedkadous Aug 3, 2026
3b4671a
chore(porch): 1286 implement build-complete
waleedkadous Aug 3, 2026
2cb1e2f
[Spec 1286][Phase: phase_6] docs: make every JSON example parseable, …
waleedkadous Aug 3, 2026
2121b73
chore(porch): 1286 implement force-advance (safety ceiling reached at…
waleedkadous Aug 3, 2026
1e02a39
chore(porch): 1286 all plan phases complete → review
waleedkadous Aug 3, 2026
0c4d40a
[Spec 1286] docs: thread notes on phase_6 approval and review-phase e…
waleedkadous Aug 3, 2026
f002543
[Spec 1286] docs: review — lessons, architecture updates, and the two…
waleedkadous Aug 3, 2026
260a881
chore(porch): 1286 record PR #1341
waleedkadous Aug 3, 2026
9f41909
chore(porch): 1286 review build-complete
waleedkadous Aug 3, 2026
251c867
[Spec 1286] fix: bidirectional SDK enum binding; correct review metadata
waleedkadous Aug 3, 2026
ff451bf
chore(porch): 1286 pr gate-requested
waleedkadous Aug 3, 2026
f143205
[Spec 1286] docs: thread notes on PR gate and the overclaim correction
waleedkadous Aug 3, 2026
5de39ff
[Spec 1286] docs: record architect-required confirming codex pass (AP…
waleedkadous Aug 3, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
193 changes: 191 additions & 2 deletions codev-skeleton/resources/commands/consult.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,28 @@ consult stats [options]
-m, --model <model> Model to use (required for all modes except stats)
```

## Model Selection Options

```
--model-id <id> Override the provider model id for THIS invocation
```

`-m/--model` picks the **lane** (`claude`, `codex`, `gemini`, `hermes`); `--model-id` picks the
**model that lane runs**. The two are independent — see [Configuration](#configuration) for setting
an id persistently instead.

```bash
consult -m codex --model-id gpt-5.6-sol --prompt "Review this design"
```

- **Precedence**: `--model-id` > `consult.models.<lane>` > the lane's shipped default.
- **Supported lanes**: `claude`, `codex`, `gemini`. Using it with `hermes` is an **error**, not a
silent no-op — `hermes chat -q` has no model selector, so accepting the flag there would mean
ignoring it.
- **Validation is syntax-only.** Whether the id exists is the provider's call; a rejection fails
loudly with no fallback to the default. See
[the fail-fast contract](#the-fail-fast-contract-and-where-it-stops).

## Models

| Model | Alias | Backend | Shipped default model id | Notes |
Expand All @@ -36,6 +58,172 @@ are still recorded, but `cost_usd` is stored as `null` rather than billed at som
rates. Only OpenAI's standard pricing tier is modelled — costs for consultations large enough to
enter the long-context tier are under-reported.

Supply rates for a model Codev doesn't know with [`consult.pricing.codex`](#consultpricingcodex).

## Configuration

Everything below lives in `.codev/config.json` and flows through the standard five-layer config
stack, lowest priority to highest:

1. built-in defaults
2. `<cache>/config.json` — remote framework base config
3. `~/.codev/config.json` — global, per-user, across all projects
4. `.codev/config.json` — project, checked in
5. `.codev/config.local.json` — project, per-engineer, gitignored

So any key can be set globally and narrowed per project, and an individual engineer can override
either without touching a checked-in file.

**Two independent axes**, easy to confuse:

| Axis | Key | Answers |
|------|-----|---------|
| *Which model* a lane runs | `consult.models` | "run `claude-opus-5` on the claude lane" |
| *Which lanes* run at all | `porch.consultation.*` | "review PIR with two lanes, not three" |

### `consult.models`

Per-lane model id. Absent → the shipped default in the [Models](#models) table. Outranked for a
single invocation by [`--model-id`](#model-selection-options).

```json
{ "consult": { "models": { "claude": "claude-opus-5", "codex": "gpt-5.6-sol" } } }
```

Valid lanes: `claude`, `codex`, `gemini`. **`hermes` is rejected** — it is invoked as
`hermes chat -q` and exposes no model selector, so configuring one would silently do nothing.
(`hermes` remains valid in `porch.consultation` lane lists; the two key spaces differ on purpose.)

The `gemini` lane passes the id to `agy --model`, so the id space is agy's, not Google's API's.

### `consult.reasoningEffort`

```json
{ "consult": { "reasoningEffort": { "codex": "high" } } }
```

Only `codex` exposes this. Values: `minimal`, `low`, `medium`, `high`, `xhigh` (default `medium`).
Unlike model ids, this **is** a closed set Codev validates locally — see the asymmetry below.

### `consult.pricing.codex`

Per-1M-token rates for the codex lane. Set this when Codev has no rates for the model you run
(otherwise `cost_usd` is `null`), or to correct rates that have gone stale.

**It outranks the shipped rate table for every model, not only unknown ones** — once set, it is
used for whatever the codex lane runs, so it is worth revisiting if you later change the model.

```json
{ "consult": { "pricing": { "codex": { "inputPer1M": 5.00, "cachedInputPer1M": 0.50, "outputPer1M": 30.00 } } } }
```

> **Take the numbers from the provider, not from here.** Those are the rates Codev ships for
> `gpt-5.6-sol` at the time of writing, shown so the shape is concrete — they are not right for
> whatever model you are configuring, and published rates change. Copying a plausible-looking wrong
> rate produces a confidently wrong cost, which is the exact failure this key exists to prevent;
> a `null` cost is the better outcome of the two.

- **`codex` is the only accepted lane.** Claude reports its own cost directly and the gemini/agy
lane reports no usage data at all, so a pricing override for either would be inert. Any other
lane key is an error.
- **All three rates are required together**, and each must be a finite, non-negative number. A
partial object is an error, not a half-priced estimate: defaulting any one rate to a stale
built-in would reintroduce exactly the wrong-cost problem this override exists to fix.

### `porch.consultation` — which lanes run

Lane lists accept a single name (`"codex"`), an array (`["codex", "claude"]`), or a whole-value
special mode: `"none"` (skip consultation) or `"parent"` (emit a gate for the architect instead).
An **empty array is rejected** — use `"none"`, so there is exactly one way to say it.

`models` is the workspace-wide default, `modelsByType` narrows by review type, and `byProtocol`
scopes either of those to one protocol:

```json
{
"porch": {
"consultation": {
"models": ["gemini", "codex", "claude"],
"modelsByType": { "pr": ["codex", "claude"] },
"byProtocol": {
"pir": {
"models": ["gemini", "codex"],
"modelsByType": { "impl": ["codex"] }
}
}
}
}
}
```

Review-type keys are the protocol's own `verify.type` values (`spec`, `plan`, `impl`, `pr`, …);
protocol keys are protocol names, and aliases are canonicalized so `byProtocol.spider` matches a
project running as `spir`. Unknown keys in either space are **errors, not warnings** — a typo that
merely warned would silently leave you on the defaults you were trying to override.

#### Precedence

Highest first. The first level that is present wins outright; levels do not merge.

1. `porch.consultation.byProtocol[<protocol>].modelsByType[<type>]`
2. `porch.consultation.byProtocol[<protocol>].models`
3. `porch.consultation.modelsByType[<type>]`
4. `porch.consultation.models`
5. the protocol's own `verify.models` (i.e. no config at all)

Both `porch next` and `porch done` resolve through this one ladder, so the lanes porch asks you to
run are exactly the lanes it will require review files for.

#### Worked example: keeping PIR cheap while widening the default

PIR is deliberately a 2-lane (CMAP-2) protocol. A workspace-wide 3-lane default silently inflates
it, because config outranks protocol. Scope PIR back down explicitly:

```json
{
"porch": {
"consultation": {
"models": ["gemini", "codex", "claude"],
"byProtocol": { "pir": { "models": ["gemini", "codex"] } }
}
}
}
```

`["gemini", "codex"]` is PIR's own shipped pair, so this restores exactly what the protocol declares
rather than substituting a different two. SPIR and ASPIR reviews run three lanes; PIR runs two.
Without the `byProtocol` entry, PIR would run three and cost 50% more per review with no change to
the protocol file.

### The fail-fast contract, and where it stops

Config errors are raised when config is **loaded** — before any consultation starts — and name the
offending key and the valid alternatives. Nothing falls back to a default on error.

**The asymmetry worth knowing about:** these two are validated very differently.

| | Validated by | When you find out |
|---|---|---|
| `reasoningEffort` | **Codev**, against a closed enum | Config load, before anything runs |
| Model ids | **The provider** | When the lane runs |

Codev checks a model id's *syntax* only (ASCII alphanumerics plus `. _ : / @ + -`, 1–200 characters,
no leading punctuation) — never its existence. **There is no allowlist of model ids anywhere in
Codev, by design**: a new model must work the day the provider ships it, without a Codev release.

So a typo'd model id is not caught at config time. It reaches the backend, which rejects it; that
lane exits non-zero, the provider's error text is surfaced, the config key that supplied the id is
named, and **no review file is written** — so porch cannot advance on a lane that never ran. What
you do *not* get is a silent substitution of the default model.

One deliberate exception: a `gemini` lane **with no model id resolved** still skips non-blockingly
when `agy` is missing or unauthenticated (consultation is best-effort there). Once an id *is*
resolved — from either `consult.models.gemini` **or** `--model-id` — a rejected model becomes a hard
failure for that lane, because you asked for a specific model and did not get it. What still skips
rather than fails, even with an id, are causes that are not the model's fault: `agy` absent,
unauthenticated, timed out, killed by a signal, or exiting **successfully** having produced no
output at all.

## Modes

### General Mode
Expand Down Expand Up @@ -90,10 +278,11 @@ consult -m codex --type integration --issue 42 --base ci
- `--base <ref>` — **`--type integration` only.** Anchor the diff on this base branch (e.g. `ci`), computed locally as `git diff origin/<base>...origin/<head>` (three-dot, merge-base anchored). Use in repos with a long-lived integration branch ahead of the default branch so the review sees only the PR's actual change, not the whole integration-over-trunk delta. Unresolvable refs fail loudly with a `git fetch` hint (no silent fallback to the local checkout). Defaults to config `consult.integrationBranch`; with neither set, the integration review uses the PR's host-recorded base (`gh pr diff`), unchanged.

**Config (`.codev/config.json`):**
```jsonc
`integrationBranch` is the repo-wide default base for `--type integration`, overridden by `--base`.

```json
{
"consult": {
// Repo-wide default base for `--type integration` (overridden by --base).
"integrationBranch": "ci"
}
}
Expand Down
Loading
Loading