Integration: Monitor scheduled jobs in Sentry

[AkhileshNegi]

Is your feature request related to a problem?
We currently use Sentry for observability. They also support setting up a cron monitor specifically designed for scheduled jobs, where it tracks whether a task checks in on its expected schedule and alerts if it misses, fails, or runs too long

Describe the solution you'd like
Integrate Sentry's cron monitor to observe scheduled jobs and alerts for missed or failed tasks.

[AkhileshNegi]

Alert types from a Sentry Cron Monitor:

1. Missed check-in

• Trigger: No "in_progress" check-in arrives within checkin_margin minutes of the expected time (config had checkin_margin: 2).
• Meaning: The job didn't start on schedule. Usually the invoker script crashed, container is down, network blocked, or auth failed
  before reaching the endpoint.
• Where to look: scripts/python/invoke-cron.py logs, the container/pod running it, and upstream networking.

2. Failed check-in

• Trigger: Job reported status=error (decorator catches unhandled exceptions and marks failure).
• Meaning: Job ran but threw. Note: your endpoint currently catches exceptions and returns {"status": "error"} — Sentry sees this as a
  success. To make Sentry see it as failed, either remove the try/except or re-raise.
• Where to look: Linked error event in Sentry (stack trace attached), plus [evaluation_cron_job] logs.

3. Timeout (max runtime exceeded)

• Trigger: max_runtime: 10 means no "ok" check-in within 10 minutes of "in_progress".
• Meaning: Job is stuck — hung DB query, blocked HTTP call to OpenAI/Langfuse, or a processing loop that never completes.
• Where to look: DB connection pool logs, LLM API latency, and whether the worker/process is alive.

4. Failure threshold breached

• Trigger: failure_issue_threshold: 2 — two consecutive failures/missed/timeouts in a row create a Sentry Issue (not just a blip).
• Meaning: Not a flake — a real ongoing problem. Pages via your Sentry alert rules.

5. Recovery

• Trigger: recovery_threshold: 1 — one successful check-in resolves the issue.
• Meaning: Auto-closes the issue. You'll see the recovery time in the monitor timeline.

[AkhileshNegi]

Overhead of @sentry_sdk.monitor on the endpoint:

Per invocation (every 5 min):

• 2 HTTP POST requests to Sentry's check-in API — one "in_progress" at start, one "ok"/"error" at end

• Requests are non-blocking (fired via Sentry's background worker thread), so they don't add latency to the endpoint response

• Payload is tiny (~200 bytes JSON per check-in)

• CPU cost: negligible — just timestamp capture and a dict serialization

• If Sentry's API is unreachable, the background worker drops the check-in silently. No impact on your endpoint.

Failure modes to know:

• If the worker thread queue fills (default 100 events), check-ins get dropped — unlikely at this volume