feat(prompts): add capture_errors option for error tracking

[andrewm4894]

Replaces #427 — rebased on current main to resolve conflicts. The old branch couldn't be force-pushed due to a signed-commits ruleset that flags historical unsigned release commits (v7.9.2–v7.9.8) as "new" on force-push/merge.

Exploratory. As a user using prompts in my app, if prompt fetches started failing and PostHog didn't surface it in error tracking, I'd be pretty upset — regardless of whether the root cause is on PostHog's side or mine. This adds an opt-in path for that.

Summary

• Adds capture_errors parameter to Prompts.__init__() (default False, fully backward compatible)
• When True and a PostHog client is provided, prompt fetch failures are reported via client.capture_exception() before the existing fallback logic runs (stale cache → code fallback → re-raise)
• If capture_exception() itself fails, it's silently swallowed and never affects prompt resolution
• Guarded with hasattr(client, "capture_exception") check (addresses Greptile review on feat(prompts): add capture_errors option for error tracking #427)

Motivation

Failed prompt fetches are almost always actionable by the app developer:

• 404 — typo in the prompt name, or someone deleted it in the UI
• 403 — personal_api_key lost permissions
• Auth errors — stale/rotated key
• Network/timeouts — infra issue worth monitoring

Today these are silently handled — the caller gets a fallback string and the error is only logged at WARNING level. With capture_errors=True, they surface in PostHog error tracking, making it easy to spot API issues, auth problems, or missing prompts without configuring Python logging.

Usage

from posthog import Posthog
from posthog.ai.prompts import Prompts

posthog = Posthog('phc_xxx', host='https://us.posthog.com', personal_api_key='phx_xxx')
prompts = Prompts(posthog, capture_errors=True)

# Fetch failures now show up in PostHog error tracking
template = prompts.get('my-prompt', fallback='default prompt')

Changes

• posthog/ai/prompts.py — new capture_errors param, _maybe_capture_error() helper with hasattr guard
• posthog/test/ai/test_prompts.py — 7 new tests covering all branches
• .sampo/changesets/prompts-capture-errors.md — sampo changeset for release (per RELEASING.md)

Test plan

• capture_exception called on fetch failure with fallback
• capture_exception called on fetch failure with stale cache
• capture_exception called when error is re-raised (no fallback/cache)
• NOT called when capture_errors=False (default)
• NOT called when no client provided (direct personal_api_key mode)
• NOT called on successful fetch
• capture_exception failure does not affect fallback resolution
• All 58 existing prompt tests pass unchanged

[github-actions]

posthog-python Compliance Report

Date: 2026-04-17 10:09:41 UTC
Duration: 160015ms

✅ All Tests Passed!

30/30 tests passed

---

Capture Tests

✅ 29/29 tests passed

View Details

Test	Status	Duration
Format Validation.Event Has Required Fields	✅	517ms
Format Validation.Event Has Uuid	✅	1507ms
Format Validation.Event Has Lib Properties	✅	1507ms
Format Validation.Distinct Id Is String	✅	1505ms
Format Validation.Token Is Present	✅	1505ms
Format Validation.Custom Properties Preserved	✅	1506ms
Format Validation.Event Has Timestamp	✅	1506ms
Retry Behavior.Retries On 503	✅	9520ms
Retry Behavior.Does Not Retry On 400	✅	3506ms
Retry Behavior.Does Not Retry On 401	✅	3506ms
Retry Behavior.Respects Retry After Header	✅	9515ms
Retry Behavior.Implements Backoff	✅	23530ms
Retry Behavior.Retries On 500	✅	7503ms
Retry Behavior.Retries On 502	✅	7514ms
Retry Behavior.Retries On 504	✅	7507ms
Retry Behavior.Max Retries Respected	✅	23522ms
Deduplication.Generates Unique Uuids	✅	1508ms
Deduplication.Preserves Uuid On Retry	✅	7510ms
Deduplication.Preserves Uuid And Timestamp On Retry	✅	14526ms
Deduplication.Preserves Uuid And Timestamp On Batch Retry	✅	7509ms
Deduplication.No Duplicate Events In Batch	✅	1504ms
Deduplication.Different Events Have Different Uuids	✅	1507ms
Compression.Sends Gzip When Enabled	✅	1507ms
Batch Format.Uses Proper Batch Structure	✅	1506ms
Batch Format.Flush With No Events Sends Nothing	✅	1004ms
Batch Format.Multiple Events Batched Together	✅	1505ms
Error Handling.Does Not Retry On 403	✅	3509ms
Error Handling.Does Not Retry On 413	✅	3506ms
Error Handling.Retries On 408	✅	7515ms

Feature_Flags Tests

✅ 1/1 tests passed

View Details

Test	Status	Duration
Request Payload.Request With Person Properties Device Id	✅	517ms

[greptile-apps]

Prompt To Fix All With AI

This is a comment left during a code review.
Path: posthog/test/ai/test_prompts.py
Line: 926-936

Comment:
**Dead mock — wrong error path exercised**

`mock_get.side_effect` is never triggered here because `Prompts` is constructed without a `project_api_key`, so `_fetch_prompt_from_api` raises `"project_api_key is required"` before the HTTP call is ever made. The test is exercising a config-error path rather than the intended "network error with no client" path, and the `@patch` decorator and `mock_get` setup are superfluous.

To test the documented scenario, supply `project_api_key` (mirroring `create_mock_posthog`'s defaults) so the mock HTTP error is actually reached.

How can I resolve this? If you propose a fix, please make it concise.

---

This is a comment left during a code review.
Path: posthog/test/ai/test_prompts.py
Line: 851-964

Comment:
**Prefer parametrised tests**

The three "NOT called" scenarios (`capture_errors=False`, no client, successful fetch) and the two basic "called-once" scenarios (with fallback, with re-raise) share nearly identical structure and could each be collapsed into a single `@parameterized.expand` test, matching the pattern already used elsewhere in this file (e.g. `test_handle_404_response`, `test_deprecation_warning_count`).

**Context Used:** Do not attempt to comment on incorrect alphabetica... ([source](https://app.greptile.com/review/custom-context?memory=instruction-0))

How can I resolve this? If you propose a fix, please make it concise.

_{Reviews (1): Last reviewed commit: "feat(prompts): add capture_errors option..." | Re-trigger Greptile}

[andrewm4894]

FYI — did a bit of research into SDK + error tracking conventions while thinking about this one. Sharing in case it's useful for anyone else looking at this later.

posthog-python SDK:

• No existing precedent for SDK self-reporting its own internal failures — all capture_exception call sites wrap user code (decorators, Django middleware, LangChain LLM-call errors).
• APIError(Exception) with a status field already exists in posthog/request.py:357 — prompts.py currently raises plain Exception with formatted strings, so if we ever want to filter by status we'd want typed exceptions.
• posthog-js has the same pattern — captureException is the public API + browser error/unhandledrejection autocapture, nothing internal.

Error tracking product:

• Suppression rules exist (ErrorTrackingSuppressionRule) — users can drop matching events at ingest with a configurable sampling rate.
• Fingerprinting dedupes identical exceptions into one issue (SHA512 hash over exception contents), so e.g. 10k identical 404s = 1 issue with 10k events.
• Billing is per-event (EXCEPTIONS quota) rather than per-issue.
• No dedicated "SDK diagnostics" or prompt-specific category/template.

Leaving it as "capture everything when the user opts in" for now — feels like the simplest starting point. If it turns out to be noisy in practice we can always add an opt-in filter later (e.g. only capture 4xx/config errors and skip 5xx/network/timeout, or a capture_errors="actionable" tri-state).

Also threaded prompt_name / prompt_version / posthog_host / $lib_feature as properties on the captured exception so suppression/grouping rules can filter on structured metadata rather than string-matching exception messages.