pika-news: fix stuck pending tasks and improve reliability

[justinmoon]

Summary

• Fix stale generating artifacts never recovering after service restart by adding startup recovery
• Fix Claude CLI tool usage causing error_max_turns by passing --tools "" and -p
• Fix JSON extraction failing when model prefixes output with prose before fenced JSON
• Add structured poll/worker logging and better error diagnostics

Test plan

• 15 unit tests pass (including 3 new tests for extraction and recovery)
• Deployed to pika-build and verified: consecutive cycles show ready=1, failed=0, retry=0
• DB went from 21→23 ready items after deploy, pending items steadily processing
• Public site at news.pikachat.org serving correctly

🤖 Generated with Claude Code

---

Summary by CodeRabbit

• Bug Fixes

  • Fixed artifacts becoming stuck in generating state after unclean shutdowns—system now automatically recovers them on startup.
  • Improved resilience to Claude API timeouts and errors with enhanced retry handling.
  • Enhanced JSON parsing robustness to handle various Claude API response formats.

• Chores

  • Improved internal logging for debugging poll and worker operations.

[coderabbitai]

📝 Walkthrough

Walkthrough

The changes introduce a startup recovery mechanism to reset artifacts stuck in a "generating" state from unclean shutdowns, enhance Claude CLI resilience with improved error handling and JSON extraction logic, and refactor logging in the web worker loop with pattern matching and conditional output.

Changes

Cohort / File(s)	Summary
Stale Artifact Recovery crates/pika-news/src/main.rs, crates/pika-news/src/storage.rs	Adds startup recovery step that calls recover_stale_generating() to reset artifacts with "generating" status to "pending", including a test verifying recovery and reclamability.
Claude CLI Resilience crates/pika-news/src/model.rs	Enhances Claude CLI invocations with -p flag and tools option; improves JSON parsing with multiple extraction strategies (fenced blocks, bare braces, outermost object fallback); extends error messages with envelope result prefixes; adds explicit handling for error_max_turns subtype and tests for JSON extraction utilities.
Worker Logging Improvements crates/pika-news/src/web.rs	Refactors error handling with pattern matching on poll_result and worker_result; adds conditional logging that includes metrics like repos_polled, head_sha_changes, ready, failed, and retry_scheduled when respective work counts exceed zero.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Poem

🐰 Recovery and resilience, a tale so fine,
Claude speaks clearer in JSON's design,
Stale artifacts wake from their slumber deep,
The worker logs metrics we proudly keep,
A healthier startup, from mess to sublime!

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 57.14% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately summarizes the main changes: fixing stuck pending tasks through startup recovery (recover_stale_generating) and improving reliability through Claude CLI fixes and robust JSON extraction.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

• 📝 Generate docstrings (stacked PR)
• 📝 Generate docstrings (commit on current branch)

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch news-fix

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[devin-ai-integration]

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no potential bugs to report.

View in Devin Review to see 5 additional findings.

[coderabbitai]

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

crates/pika-news/src/model.rs (1)

112-121: ⚠️ Potential issue | 🟠 Major

Disable tools in chat_with_session to match behavior in generate_tutorial.

The generate_tutorial function uses --tools "" to disable tools, but chat_with_session omits this flag entirely. This inconsistency means the two code paths will behave differently—one with tools disabled, the other with tools enabled by default. This divergence can cause tool-related failures to manifest unpredictably in chat responses.

Proposed fix

     let stdout = run_claude_cli(
         &[
             "-p",
             "-r",
             base_session_id,
+            "--tools",
+            "",
             "--output-format",
             "json",
             "--max-turns",
             "1",
         ],

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@crates/pika-news/src/model.rs` around lines 112 - 121, chat_with_session is
calling run_claude_cli without the --tools "" flag, causing tools to be enabled
by default and diverging from generate_tutorial; update the chat_with_session
call site (the run_claude_cli invocation) to include the --tools and
empty-string arguments (i.e. "--tools", "") so tools are explicitly disabled,
ensuring consistent behavior with generate_tutorial and preventing tool-related
differences in chat responses.

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Outside diff comments:
In `@crates/pika-news/src/model.rs`:
- Around line 112-121: chat_with_session is calling run_claude_cli without the
--tools "" flag, causing tools to be enabled by default and diverging from
generate_tutorial; update the chat_with_session call site (the run_claude_cli
invocation) to include the --tools and empty-string arguments (i.e. "--tools",
"") so tools are explicitly disabled, ensuring consistent behavior with
generate_tutorial and preventing tool-related differences in chat responses.

---

ℹ️ Review info

Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 5aa95897-2f82-4c5e-b6bc-ac5ac589f5a6

📥 Commits

Reviewing files that changed from the base of the PR and between c88b0ac and 8f9f8f6.

📒 Files selected for processing (4)

• crates/pika-news/src/main.rs
• crates/pika-news/src/model.rs
• crates/pika-news/src/storage.rs
• crates/pika-news/src/web.rs