fix: harden workflow archive DB retries

[zhaohao1004]

Summary

This PR hardens the workflow run archive command against transient DB disconnects during archive planning and bundle processing.

What changed

• Added a shared workflow archive DB retry helper that only retries SQLAlchemy/psycopg operational disconnects.
• Switched archive plan and tenant-prefix resolution to explicit short-lived sessions with bounded retry.
• Changed tenant plan resolution failure to raise ClickException, so Kubernetes CronJobs do not mark failed planning as success.
• Kept post-archive scoped-session cleanup best-effort so a completed archive is not turned into a non-zero exit by stale cleanup.
• Added focused tests for DB retry, non-DB error classification, plan/count retry, and ClickException behavior.

Root cause

The archive command could still fail on stale DB connections in planning paths, while some planning failures were printed and returned from the Click command, causing a successful process exit. The retry classification also matched generic network-looking messages without first verifying that the exception was DB-related.

Validation

• uv run --project api --group dev ruff check api/commands/retention.py api/services/retention/workflow_run/archive_paid_plan_workflow_run.py api/services/retention/workflow_run/db_retry.py api/tests/integration_tests/services/retention/test_workflow_run_archiver.py api/tests/unit_tests/commands/test_archive_workflow_runs.py
• uv run --project api --group dev pytest api/tests/integration_tests/services/retention/test_workflow_run_archiver.py api/tests/unit_tests/commands/test_archive_workflow_runs.py
• uv run --project api python -m py_compile api/commands/retention.py api/services/retention/workflow_run/archive_paid_plan_workflow_run.py api/services/retention/workflow_run/db_retry.py api/tests/integration_tests/services/retention/test_workflow_run_archiver.py api/tests/unit_tests/commands/test_archive_workflow_runs.py

[github-actions]

Pyrefly Diff

base → PR

--- /tmp/pyrefly_base.txt	2026-07-06 10:28:23.067683305 +0000
+++ /tmp/pyrefly_pr.txt	2026-07-06 10:28:13.480704387 +0000
@@ -559,9 +559,9 @@
 ERROR `unpatch` may be uninitialized [unbound-name]
   --> tests/integration_tests/plugin/__mock/http.py:67:9
 ERROR Argument `FakeArchiveStorage` is not assignable to parameter `storage` with type `ArchiveStorage | None` in function `services.retention.workflow_run.archive_paid_plan_workflow_run.WorkflowRunArchiver._archive_bundle` [bad-argument-type]
-   --> tests/integration_tests/services/retention/test_workflow_run_archiver.py:412:56
+   --> tests/integration_tests/services/retention/test_workflow_run_archiver.py:530:56
 ERROR Argument `FakeArchiveStorage` is not assignable to parameter `storage` with type `ArchiveStorage | None` in function `services.retention.workflow_run.archive_paid_plan_workflow_run.WorkflowRunArchiver._archive_bundle` [bad-argument-type]
-   --> tests/integration_tests/services/retention/test_workflow_run_archiver.py:437:60
+   --> tests/integration_tests/services/retention/test_workflow_run_archiver.py:555:60
 ERROR Argument `_stub_resolver._Resolver` is not assignable to parameter `binding_resolver` with type `WorkflowAgentBindingResolver | None` in function `services.workflow.node_output_inspector_service.NodeOutputInspectorService.__init__` [bad-argument-type]
    --> tests/integration_tests/services/test_node_output_inspector_service.py:231:59
 ERROR Argument `_stub_resolver._Resolver` is not assignable to parameter `binding_resolver` with type `WorkflowAgentBindingResolver | None` in function `services.workflow.node_output_inspector_service.NodeOutputInspectorService.__init__` [bad-argument-type]
@@ -1855,6 +1855,8 @@
 ERROR `None` is not subscriptable [unsupported-operation]
   --> tests/unit_tests/clients/agent_backend/test_event_adapter.py:54:12
 ERROR Expected a callable, got `None` [not-callable]
+   --> tests/unit_tests/commands/test_archive_workflow_runs.py:124:9
+ERROR Expected a callable, got `None` [not-callable]
   --> tests/unit_tests/commands/test_clean_expired_messages.py:33:9
 ERROR Expected a callable, got `None` [not-callable]
   --> tests/unit_tests/commands/test_clean_expired_messages.py:63:9

[github-actions]

Pyrefly Type Coverage

Metric	Base	PR	Delta
Type coverage	52.39%	52.38%	-0.01%
Strict coverage	51.90%	51.89%	-0.01%
Typed symbols	32,092	32,100	+8
Untyped symbols	29,442	29,457	+15
Modules	2970	2972	+2

[codecov]

Codecov Report

❌ Patch coverage is 55.55556% with 56 lines in your changes missing coverage. Please review.
✅ Project coverage is 85.40%. Comparing base (6dff7c0) to head (e5152d6).
⚠️ Report is 88 commits behind head on main.

Files with missing lines	Patch %	Lines
...ion/workflow_run/archive_paid_plan_workflow_run.py	35.13%	24 Missing ⚠️
api/commands/retention.py	67.24%	18 Missing and 1 partial ⚠️
api/services/retention/workflow_run/db_retry.py	58.06%	10 Missing and 3 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #38170      +/-   ##
==========================================
+ Coverage   85.29%   85.40%   +0.10%     
==========================================
  Files        5004     4989      -15     
  Lines      264931   262644    -2287     
  Branches    50365    49421     -944     
==========================================
- Hits       225984   224304    -1680     
+ Misses      34510    33875     -635     
- Partials     4437     4465      +28     

Flag	Coverage Δ
api	85.57% <55.55%> (+0.02%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.