RAG hardening — remove silent fails + add preflight

[cipher813]

Summary

Applies the PR #24/#25/#28 pattern to the Saturday RAG ingestion path. The shell script was silently swallowing failures in all 5 pipelines and then sending a completion email with a hardcoded `status: ok` — meaning "RAG Weekly Ingestion Complete" was a lie whenever any step failed.

Changes

`rag/pipelines/run_weekly_ingestion.sh`

• Removed `|| echo "WARNING: ... (non-fatal)"` from all 5 ingestion steps, the CloudWatch heartbeat, and the completion email. `set -e` was already active but these swallowers defeated it.
• Removed the runtime `if [ -n "$FINNHUB_API_KEY" ]; ... else echo SKIPPED fi` branch. All required env vars are hard-failed by preflight before any ingestion runs.
• Added `Step 0/5: python -m rag.preflight`.
• The hardcoded `status: ok` completion email is now truthful rather than aspirational — with `set -e` active and no swallowers, reaching the email means all 5 pipelines actually succeeded.

`rag/preflight.py` (new)

• `RAGPreflight(BasePreflight)` — composes `check_env_vars` (AWS_REGION, VOYAGE_API_KEY, FINNHUB_API_KEY, EDGAR_IDENTITY, RAG_DATABASE_URL) + `check_s3_bucket`.
• `main()` uses alpha-engine-lib's `setup_logging` with the shared `flow-doctor.yaml` path, so a preflight failure fires email + issue via the existing dispatch.

`rag/db.py`

• `is_available()`: `log.debug` → `log.warning` for the exception path.

`rag/pipelines/ingest_8k_filings.py`

• Per-URL download failure: `log.debug` → `log.warning`. Aggregate counts are still reported upstream, so no behavior change — failures are just visible now.

Dead code flagged (no change in this PR)

• `rag/db.py::is_available` — zero callers inside alpha-engine-data. Keep for now; defer deletion until cross-repo audit confirms no usage in predictor/research/backtester.

Out of scope (tracked)

• Adopt alpha-engine-lib `setup_logging` in each ingestion script's `main()` for consistent log formatting + flow-doctor capture of per-pipeline errors. Currently only `preflight.py` uses the lib. Minor follow-up.
• Date-parsing `except ValueError: continue` patterns in the ingest modules. Reviewed case-by-case — all are legitimate "skip this malformed entry" flows with aggregate counts upstream. Not silent fails.

Test plan

• `pytest tests/ --ignore=tests/integration -q` — 41 pass
• Syntax check on all modified Python files
• `bash -n run_weekly_ingestion.sh`
• Next Saturday Step Function run exercises the hardened path.
• Forced failure test on EC2: unset `FINNHUB_API_KEY` → preflight (step 0) must hard-fail, not silently skip step 3.

🤖 Generated with Claude Code