Pipeline Design 242

Design: Misleading "jq not available" warning when Claude outputs JSON object instead of array

Context

The _extract_text_from_json function in scripts/sw-loop.sh (line 555) parses Claude CLI JSON output to extract .result text for iteration logs. The function uses a first_char dispatch pattern: it reads the first byte of the output file to determine the JSON shape and route to the appropriate parser.

The bug: The original code only checked for [ (JSON array) when deciding whether to use jq. When Claude returned a JSON object ({...}) — which happens in certain Claude CLI output modes — the function skipped the jq parsing path entirely and fell through to Case 3, which emitted warn "JSON output but jq not available — using raw output". This warning was factually wrong: jq was available, the code simply didn't recognize { as valid JSON input.

Constraints:

• Bash 3.2 compatible (macOS default) — no associative arrays or advanced string ops
• The function is called on every loop iteration, so it must be fast and never crash
• jq may genuinely not be installed on some systems — the "jq not available" warning must remain for that case
• Claude CLI can return JSON in multiple shapes: arrays ([{...}]), objects ({...}), plain text, or empty output

Decision

Extend the first_char dispatch to recognize both [ and { as valid JSON, routing both to the jq parsing path. Within that path, branch on first_char to apply the correct jq expression for each shape:

Data flow (4 cases, unchanged structure):

1. Empty file — write (no output) placeholder
2. JSON array or object (jq available) — try .result, fall back to .content, then placeholder
3. JSON array or object (jq unavailable) — warn + copy raw file (now accurately reports jq absence)
4. Plain text — copy raw file

Extraction logic within Case 2:

• Arrays: .[-1].result // empty then .[].content // empty (unchanged from original)
• Objects: .result // empty then .content // empty (new)

The guard condition changed from first_char == "[" to first_char == "[" || first_char == "{". Case 3 is now only reachable when jq is genuinely not installed, making its warning message accurate.

Error handling: All jq calls use 2>/dev/null and || true to suppress parse errors on malformed JSON. If jq parses successfully but finds no .result or .content, a descriptive placeholder is written with a warn() directing the user to inspect the raw JSON file.

Alternatives Considered

1. Normalize all output to arrays before parsing — Pros: Single jq expression for both shapes, simpler control flow / Cons: Requires an extra jq invocation or sed transformation on every iteration; changes the file on disk which could affect downstream consumers; more complex error handling if the normalization itself fails on malformed input.

2. Use a single jq expression with if type == "array" conditional — Pros: One jq call instead of branching in bash / Cons: The jq expression becomes harder to read and debug (if type == "array" then .[-1].result else .result end // empty); error messages lose specificity about which shape was encountered; conflates two distinct extraction semantics into one expression. This approach was also observed in failure patterns — complex jq expressions on varied input types caused parse errors in production.

3. Always dump raw JSON, skip extraction entirely — Pros: Zero parsing risk / Cons: Defeats the purpose of the function; iteration logs would contain raw JSON blobs instead of human-readable text summaries; downstream consumers (progress.md, dashboard) expect plain text.

Implementation Plan

• Files to create: none
• Files to modify:
  • scripts/sw-loop.sh — _extract_text_from_json function (lines 575-617): extend Case 2 guard, add object-specific jq expressions
  • scripts/sw-loop-test.sh — Test 23: add 4 assertions for JSON object extraction and absence of misleading warning
• Dependencies: none (jq is an existing optional dependency)
• Risk areas:
  • Regression in array handling: The array path is unchanged code, but sharing the Case 2 entry point means a typo in the guard condition could break it. Mitigated by existing Test 21 covering array extraction.
  • Malformed JSON objects: A file starting with { but containing invalid JSON (e.g., truncated output) will fail jq parsing silently and fall through to the placeholder. This is correct behavior — same as existing array handling.
  • Performance: No impact — head -c1 and the first_char branch add negligible overhead.

Validation Criteria

• JSON object with .result field: extracts result text, no warnings
• JSON object with .content field (no .result): extracts content as fallback
• JSON object with neither field: writes descriptive placeholder, warns about missing .result
• JSON array input: behaves identically to before the change (regression check)
• "jq not available" warning does NOT appear when jq IS installed and input is {...}
• "jq not available" warning DOES appear when jq is genuinely absent
• All 69 existing loop tests continue to pass

Root Cause Analysis (Systematic Debugging)

Root Cause Hypothesis

1. Most likely: The first_char == "[" guard was written when Claude CLI only emitted JSON arrays. When the CLI started emitting single objects, the guard became an incomplete check, causing valid JSON to bypass jq. Evidence: The diff shows only the guard condition and branching within Case 2 were changed — the function structure was otherwise correct.
2. Less likely: Claude CLI behavior changed in a version update, producing objects where arrays were expected. Evidence: Historical failure patterns in failures.json document both array and object outputs, suggesting both shapes have been possible for some time.
3. Unlikely: jq itself fails on object input. Evidence: jq handles objects natively; the issue was never reaching the jq call.

Evidence Gathered

• scripts/sw-loop.sh:579 — original guard: first_char == "[" (only arrays)
• scripts/sw-loop.sh:609 — Case 3 warning triggered for objects: warn "JSON output but jq not available"
• Memory failures.json — 3 entries documenting jq parse errors on object-shaped output, confirming objects reach production
• The fix: 18 lines changed in sw-loop.sh, 45 lines added in sw-loop-test.sh

Fix Strategy

The fix addresses the root cause (incomplete guard condition) rather than the symptom (misleading warning text). Changing only the warning message would hide the real issue — valid JSON objects not being parsed, causing raw JSON in iteration logs.

Verification Plan

• Run ./scripts/sw-loop-test.sh — all 69 tests pass including 4 new assertions
• Test 23 explicitly asserts the misleading warning is absent when jq is present
• Existing Test 21 covers array extraction as a regression gate