feat(docs-pipeline): source grounding, compiler gate, and pipeline fixes

[jmanhype]

Summary

Implements the Autonomous Self-Demonstrating Simulation Engine for the DocumentationWriterBot. The pipeline now:

• Grounds writer in actual source code — GroundSourceCode reads lib/jido/agent.ex and lib/jido_action.ex from cloned repos
• Runs a deterministic compiler gate — EvaluateLivebookDraft extracts all ```elixir blocks, combines them into a .exs script, and executes in the sprite sandbox. Real compilation errors feed into the critic for targeted revision instructions
• Propagates content_metadata through the full 23-node DAG without dropping docs-specific keys at shared action boundaries

Bug Fixes (8 total)

1. Enum.find_value source grounding bug — cat file | head -n 1000 exits 0 even when cat fails (head's exit code wins), so error messages like "cat: lib/jido_action.ex: No such file" were accepted as valid content. Fixed with test -f gate + text guard.
2. Compiler gate output capture — Non-zero exit from elixir script.exs returned {:error, %Shell.Error{}} which didn't match the expected pattern, losing the actual compilation trace. Fixed with || true to always capture output.
3. Trace truncation — Mix.install output (37KB of package resolution) blew up the critic prompt payload past Sprites API limits. Now truncates to last 4KB.
4. Error detection false positive — has_error_indicators? matched bare "failed" in normal Jido runtime log lines during successful execution. Narrowed to specific Elixir error patterns.
5. Observer stop crash — rescue doesn't catch OTP :exit signals from GenServer.stop. Changed to catch :exit.
6. content_metadata propagation — Shared @pipeline_keys didn't include docs-specific keys, so shared actions dropped them. Added 10 keys to both parent and child helpers.
7. ParseContentBrief metadata — Overwrote intake metadata with empty %{} when brief had no frontmatter. Now preserves existing metadata.
8. @max_lines overflow — 1000 lines × 2 files = 58KB grounded context → 414 URI Too Long. Reduced to 250 lines.

New Files (9)

• GroundSourceCode — Extracts source code from repos for semantic grounding
• EvaluateLivebookDraft — Compiler gate: runs extracted code in sprite sandbox
• ParseContentBrief — Parses Elixir map frontmatter from content plans
• ResolveOutputPath — Resolves output path from content metadata
• EmbedInteractiveDemo — Embeds interactive demo blocks in accepted drafts
• ContentPlan — Elixir map frontmatter parser module
• Dockerfile.docs — Docker image for Linux-only erlexec builds
• mix jido_lib.github.docs.pipeline — Single content plan pipeline task
• mix jido_lib.github.docs.generate_all — Batch pipeline runner

Pipeline Test Results

All 27 DAG nodes execute end-to-end. Source grounding successfully reads both Jido.Agent (9KB) and Jido.Action (9KB). Compiler gate correctly catches compilation errors and feeds traces to critic. Writer/critic revision loop operates correctly (V1 → gate → critic → V2 → gate → final decision).

Test plan

• Pipeline runs all 27 nodes without infrastructure errors
• Source grounding reads files from correct repos (not cat error messages)
• Compiler gate captures actual compilation errors (not SYSTEM ERROR)
• Trace truncation keeps critic prompt under Sprites API limit
• Error detection doesn't false-positive on runtime log lines
• Run pipeline with --provider claude until writer produces accepted output

🤖 Generated with Claude Code

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5924ae58d2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Parse frontmatter without evaluating arbitrary Elixir

ContentPlan.parse_string!/1 executes frontmatter via Code.eval_string/1, so any content plan processed by mix jido_lib.github.docs.pipeline, generate_all, or scan_drift can run arbitrary BEAM code during parsing (for example System.cmd/3) instead of being treated as data. This is a code-execution path triggered by file contents and is unsafe in automation/CI contexts that ingest untrusted branches or PR content.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Avoid prepending a second frontmatter block to final guides

This path always prepends build_frontmatter/2 to clean_draft, but RunWriterPass now instructs both v1 and v2 drafts to start with their own %{...} frontmatter. As a result, accepted outputs systematically contain two frontmatter sections, which breaks the expected single-header Livebook/NimblePublisher format and leaves one metadata map rendered as body content.

Useful? React with 👍 / 👎.