fix(purchases): recover executions stranded in 'approved' (closes #632)

[cristim]

Summary

Fixes the P1 where an approved purchase strands permanently in approved (purchase never runs, no error, completed_at NULL) when the synchronous execution is interrupted (Lambda timeout / cold-start eviction / panic) — see #632.

Approach: recovery sweep + safe-fail (not auto-re-drive)

A new RecoverStrandedApprovals runs at the top of the existing advisory-locked ProcessScheduledPurchases scheduled task. It calls a new store method GetStaleApprovedExecutions(olderThan) (WHERE status='approved' AND updated_at < NOW() - interval, mirroring the existing GetStaleProcessingExchanges RI-exchange prior art) and drives each stranded row into terminal failed with a clear operator-readable error. Threshold is 15 min (the purchase Lambda timeout is 60s, so no in-flight run can ever be failed under itself).

Why fail, not re-run (the key decision)

Commitment creation has no idempotency token: EC2 PurchaseReservedInstancesOffering sets no ClientToken and CreateSavingsPlan has no idempotency field. Auto-re-driving a row that was interrupted after AWS created the commitment but before the row persisted would double-purchase. So the sweep never re-runs the purchase (asserted via mockFactory.AssertNotCalled); it only fails the row. The approved->failed transition uses atomic TransitionExecutionStatus (WHERE status='approved'), so a late-completing original run is never clobbered. Failed rows are visible in History (#623) and Retry-able after the operator confirms AWS state.

The idempotent auto-re-drive path is deferred (needs a ClientToken/dedupe prerequisite) and filed as a follow-up.

Test plan

• Regression test: stranded approved -> failed, purchase NOT re-executed, a fresh approved row (< threshold) untouched, a late-completed row skipped
• All 10 ProcessScheduled/Recover tests green; scheduler/server/api/analytics 1763 pass; config 497 pass; go build/go vet/golangci-lint clean; pre-commit passed

Closes #632.

Summary by CodeRabbit

• Bug Fixes

  • Executions stuck in "approved" beyond the staleness threshold are now automatically moved to "failed".

• New Features

  • Processing results now include a "recovered" metric counting approvals transitioned during recovery.
  • Scheduled purchases processing runs the recovery sweep before normal work, and recovery failures are logged without blocking processing.

• Tests

  • Added regression tests covering recovery behavior and edge cases.

[cristim]

@coderabbitai review

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 811701d2-88b1-4bd7-bc9c-e4bb83774e29

📥 Commits

Reviewing files that changed from the base of the PR and between d97e0e3 and c37467b.

📒 Files selected for processing (1)

• internal/purchase/manager.go

---

📝 Walkthrough

Walkthrough

Adds a recovery sweep for purchase executions stuck in "approved": new store interface method and Postgres query, Manager.RecoverStrandedApprovals with recovered-count tracking, handler log metric, and test/mocks updates plus unit tests for recovery behaviors.

Changes

Stranded Approval Recovery

Layer / File(s)	Summary
Interface Contract and Database Layer internal/config/interfaces.go, internal/config/store_postgres.go	New GetStaleApprovedExecutions(ctx, olderThan) on StoreInterface; Postgres implementation selects purchase_executions with status='approved' and updated_at older than the provided duration and returns via existing query helper.
Manager Recovery Logic and Integration internal/purchase/manager.go	Adds ProcessResult.Recovered and RecoverStrandedApprovals(ctx) which lists stale approvals, atomically transitions eligible rows to failed with a recovery message, counts recovered rows, and invokes this sweep at the start of ProcessScheduledPurchases (errors logged, not blocking).
Handler Logging Update internal/server/handler.go	Updates handleProcessScheduledPurchases success log to include the recovered count from ProcessResult.
Test Mock Infrastructure internal/api/mocks_test.go, internal/purchase/mocks_test.go, internal/scheduler/scheduler_test.go, internal/analytics/collector_test.go, internal/server/test_helpers_test.go	Adds GetStaleApprovedExecutions(ctx, olderThan) mock/stub methods across test doubles following existing mock patterns so tests can exercise the new store call.
Manager Recovery Tests and Existing Test Updates internal/purchase/manager_test.go	Updates existing ProcessScheduledPurchases tests to expect the new store call and adds three regression tests for RecoverStrandedApprovals: successful recovery, fresh result no-op, and race where a late completion prevents clobbering.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Possibly related issues

• feat(purchases): idempotent commitment creation (ClientToken/dedupe) to enable auto-re-drive of stranded approvals #636: Related conceptually — proposes idempotency/dedupe changes to allow re-driving recoveries instead of failing; touches the broader recovery/idempotency design.
• bug(purchases): approve strands execution in 'approved' (purchase never runs, no error) on interrupted sync execution #632: Directly related — this PR implements the recovery sweep for stranded "approved" executions described in the issue.

Suggested labels

priority/p2, severity/medium, urgency/this-quarter, impact/many

Poem

🐰 I hop through queues where approvals stall,
Finding rows that time forgot at all.
With careful checks and an atomic sweep,
I mark them failed so pipelines sleep.
Recovery done — the system hums; I nap, content and spry. 🌿

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 30.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and accurately describes the main change: adding recovery functionality for executions stuck in 'approved' status, and directly addresses the issue (#632) being fixed.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/stranded-approval-recovery

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

✅ Actions performed

Review triggered.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@internal/purchase/manager.go`:
- Around line 190-194: The current loop treats every error returned from
TransitionExecutionStatus as a benign race and continues, which hides real
DB/query failures; change the handling in the block that checks txErr (around
the call to TransitionExecutionStatus for exec.ExecutionID) to distinguish
sentinel "already transitioned"/"not found" errors from other failures: use
errors.Is (or compare against your package sentinel errors like
ErrAlreadyTransitioned/ErrNotFound or sql.ErrNoRows) and only call logging.Warnf
and continue for those specific cases, but for any other txErr log it as an
error and return/propagate it to fail the sweep so stranded rows aren’t silently
skipped.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: bde7204b-d1c9-4d0b-935b-4a6377222ea9

📥 Commits

Reviewing files that changed from the base of the PR and between 9ecab71 and d97e0e3.

📒 Files selected for processing (10)

• internal/analytics/collector_test.go
• internal/api/mocks_test.go
• internal/config/interfaces.go
• internal/config/store_postgres.go
• internal/purchase/manager.go
• internal/purchase/manager_test.go
• internal/purchase/mocks_test.go
• internal/scheduler/scheduler_test.go
• internal/server/handler.go
• internal/server/test_helpers_test.go

[cristim]

@coderabbitai review

[coderabbitai]

✅ Actions performed

Review triggered.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[cristim]

@coderabbitai review

[coderabbitai]

✅ Actions performed

Review triggered.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.