[bug] LLM judge silently falls back to simple judge when endpoint unreachable — artifact is indistinguishable from a real LLM verdict

[dogkeeper886]

Context

Surfaced while forensically tracing CI run 24659380207 — the first dual-mode dispatch on the self-hosted runner. The per-test JSON artifacts reported `llmJudge.reason` === `simpleJudge.reason` verbatim for every test, and `summary.json` claimed the LLM judge passed 1 and failed 1 — identical to simpleJudge's counters. It looked like the LLM agreed with simple.

The job's stderr (which the artifact doesn't preserve) told the real story:

```
[JUDGE] Running simple judge...
[JUDGE] Running LLM judge...
[WARN] LLM judge not available, using simple judge results
```

The LLM never ran. The artifact had no signal that this was the case.

Reproduction

Deterministic. Point the framework at an unreachable endpoint:

```
bash cicd/scripts/run-tests.sh --suite smoke --id TC-SMOKE-002 \ --judge-url http://127.0.0.1:1 --format json > /tmp/out.json 2> /tmp/err.log

grep -E '\[JUDGE\]|\[WARN\]|\[LLM\]' /tmp/err.log

[JUDGE] Running simple judge...

[JUDGE] Running LLM judge...

[WARN] LLM judge not available, using simple judge results

jq '{simple: .simpleJudge.reason, llm: .llmJudge.reason}' \ cicd/results/$(ls -1t cicd/results/ | head -1)/TC-SMOKE-002.json

{

"simple": "All steps passed with exit code 0, patterns matched, no errors",

"llm": "All steps passed with exit code 0, patterns matched, no errors"

}

```

The two reason strings are byte-identical because the fallback literally copies `simpleJudgments`.

Code location

`cicd/tests/src/cli.ts:139-154`:

```ts
let llmJudgments = simpleJudgments.map((j) => ({
...j,
reason: config.noLlm ? 'LLM judge disabled' : j.reason,
}));

if (!config.noLlm) {
process.stderr.write('[JUDGE] Running LLM judge...\n');
const llmJudge = new LLMJudge(config.judgeUrl, config.judgeModel);

const available = await llmJudge.isAvailable();
if (available) {
llmJudgments = await llmJudge.judgeResults(results);
await llmJudge.unloadModel();
} else {
process.stderr.write('[WARN] LLM judge not available, using simple judge results\n');
}
}
```

And `cicd/tests/src/judge/llm-judge.ts:27-36`:

```ts
async isAvailable(): Promise {
try {
const response = await axios.get(`${this.ollamaUrl}/api/tags`, { timeout: 5000 });
return response.status === 200;
} catch {
return false;
}
}
```

Two problems compound each other:

• `isAvailable()` swallows the axios error — we can't tell `ECONNREFUSED` from `ENOTFOUND` from a 5s timeout after the fact. Makes triage blind.
• `cli.ts` writes `simpleJudgments` as `llmJudgments` when unavailable. The downstream JSON reporter writes them indistinguishably from real LLM output.

Why it matters

The whole point of dual-judge mode is that a disagreement between simple and LLM is a high-signal human-triage flag. When LLM silently clones simple, dual mode becomes no-op mode — but reports success. Regressions the LLM is uniquely positioned to catch get masked as passing.

Also: when this is paired with CLAUDE.md's statement that "LLM judge configuration lives in `cicd/tests/.env`", anyone debugging a mysterious lack of LLM coverage has no signal in the artifact to guide them.

Proposed fix

1. Log the actual error in `isAvailable()`. Replace `catch {}` with `catch (e) { log with code and message }`. Costs nothing; saves hours.
2. When unavailable, write a distinct verdict rather than copying simpleJudgments. Options:
   • `llmJudgments[i] = { testId, pass: false, reason: 'LLM judge unavailable: ${errorCode}', evidence: '' }` — fails the test (pessimistic; forces attention).
   • Or `pass: simpleJudgment.pass, reason: '[LLM unavailable — falling back to simple judge] ${simple.reason}', evidenceGrounded: false` — carries simple's verdict but makes the fallback visible.
3. Annotate `summary.json` with `llm.availability: 'ok' | 'unreachable' | 'disabled'` so downstream consumers can check without parsing reasons.

Pick (2b) + (3) as the least-surprise default: simple judge's verdict still drives the run, but every artifact and the summary make clear the LLM didn't contribute. Pessimistic-fail mode (2a) is defensible but changes the pass/fail contract of the run.

Acceptance criteria

• An artifact from a dual-mode run with an unreachable LLM endpoint is distinguishable from one where the LLM ran and agreed. A string match or jq filter should flag it.
• `summary.json` exposes LLM availability as a first-class field.
• `isAvailable()` writes the error code/message to stderr when it fails, so CI logs show why.

Out of scope

• Making `isAvailable()` more aggressive (retry, warmup call, etc.) — separate decision.
• Changing whether fallback passes or fails the run — this issue is about visibility first; the verdict-policy question is (2a) vs (2b) and can land later.

Related

• [chore] Set up a self-hosted GitHub Actions runner on the Ollama network #22 / PR ci(workflows): add runner input; self-hosted runner setup docs — #22 #24 — self-hosted runner wiring that made dispatching dual mode possible
• [bug] run-tests.sh lifecycle echoes corrupt --format json when stdout is captured #25 — stdout corruption bug found in the same trace session

[dogkeeper886]

Design update — tiered fallback, not silent

Revisiting this after seeing the simple-mode pipeline land green end-to-end (run 24665554338 after #25 / #29 fixes). The "use simple judge results as llmJudgments" default is worse than it looks: dual mode becomes no-op mode without any signal, and we just proved the infra is happy — so any future LLM regression would be pinned on infra, not on a broken judge path.

Proposed revision to the fix: tiered fallback with explicit, loud signalling.

Tier 1 — configured Ollama (the happy path)

Unchanged. `LLM_JUDGE_URL` / `LLM_JUDGE_MODEL` via workflow secrets/vars or local `.env`. If `isAvailable()` succeeds, run as today.

Tier 2 — Claude CLI on the runner

If Tier 1 is unreachable (or disabled by a flag), try `claude` from PATH. The self-hosted runner user already has it — it's how the rest of this repo gets work done. Invocation shape:

```bash
echo "$PROMPT" | claude --json --print
```

(exact flags depend on the CLI version; point is, `claude` has a non-interactive mode that returns structured output)

Framework plumbing: a new `ClaudeCliJudge` class with the same `judge(test, result)` contract as `LLMJudge`. The existing buildPrompt / parseVerdict logic can be reused — it's already provider-agnostic since we ask for a strict JSON schema in the response.

Rationale:

• No network dependency from the runner to a LAN Ollama — falls back entirely inside the runner's local environment.
• Anthropic's model is typically a sharper judge than `gemma3:4b`, so this is a quality-gain fallback, not a quality-loss one.
• Cost is real but bounded — dispatches are manual (`workflow_dispatch`), not on every push.
• Requires the runner user to be logged into `claude`. That's a one-time setup step, documented in `CI_SETUP.md`.

Tier 3 — simple judge with explicit notice (last resort)

If both Tier 1 and Tier 2 are unavailable:

• Emit a `::warning::` annotation on the GitHub job so it's visible in the run summary, not just buried in stderr.
• `llmJudgments[i].reason` = `"[LLM judge unavailable; using simple judge verdict] " + simpleJudgment.reason` — carries simple's verdict but makes the fallback visible in the artifact.
• `summary.json` gets `llm.availability: "ollama_ok" | "claude_cli_ok" | "unavailable"` + `llm.fallback_reason: "..."`.
• Overall run still passes/fails on simple judge's verdict — we don't turn an unreachable LLM into a hard failure, because infra issues shouldn't block the run. But we make very sure nobody mistakes this for a dual-mode success.

Separate observability fix (apply regardless of fallback choice)

`LLMJudge.isAvailable()` at `llm-judge.ts:27-36` needs to log the actual axios error. Replace `catch { return false }` with:

```ts
} catch (e: any) {
process.stderr.write(`[LLM] isAvailable failed: code=${e.code} message=${e.message}\n`);
return false;
}
```

An 11ms ECONNREFUSED looks identical to a 5s ETIMEDOUT when you can't see either. Without this, every future triage starts blind.

Acceptance criteria — revised

• CI dispatch with Tier 1 reachable → LLM judge runs as today; no change.
• CI dispatch with Tier 1 unreachable + `claude` CLI available on the runner → `ClaudeCliJudge` runs, artifacts are indistinguishable in quality from Tier 1 (grounded evidence, real reasons).
• CI dispatch with both unavailable → simple judge carries the run, GitHub job UI shows a warning annotation, `summary.json` exposes `llm.availability` + `llm.fallback_reason`, per-test `llmJudge.reason` is prefixed `[LLM judge unavailable; ...]`.
• `isAvailable()` logs the error code/message on failure.

Scope

Still one issue, not three — all three tiers share the same failure-visibility and schema changes. Tier 2 is the biggest ticket but the same contract as the rest.