Skip to content

feat(sf): wire replay-concordance Lambda into Saturday SF#167

Merged
cipher813 merged 1 commit into
mainfrom
feat/sf-replay-concordance-state
May 6, 2026
Merged

feat(sf): wire replay-concordance Lambda into Saturday SF#167
cipher813 merged 1 commit into
mainfrom
feat/sf-replay-concordance-state

Conversation

@cipher813
Copy link
Copy Markdown
Owner

Summary

Companion to alpha-engine-backtester #138 (concordance Lambda implementation). Inserts the Saturday SF state that fires the weekly cheap-model concordance pipeline + IAM allowlists.

SF chain after this PR

```
... → EvalJudge{Weekly,FirstSaturday} → EvalRollingMean
→ CheckSkipRationaleClustering → RationaleClustering
→ CheckSkipReplayConcordance → ReplayConcordance
→ SaturdayHealthCheck → ...
```

Independent skip-gates per observability path

Flag Effect
`skip_eval_judge: true` bypass judge only
`skip_rationale_clustering: true` bypass clustering only (rerouted from SaturdayHealthCheck → CheckSkipReplayConcordance in this PR)
`skip_replay_concordance: true` bypass concordance only

Each Catch lands at `SaturdayHealthCheck` — eval observability, failures don't halt the pipeline.

Default ReplayConcordance payload

```json
{
"end_time_iso.$": "$$.Execution.StartTime",
"target_models": ["claude-haiku-4-5"],
"window_days": 56,
"max_artifacts": 150
}
```

  • `max_artifacts=150` fits 900s Lambda timeout at ~3-5 sec / replay
  • Production weekly cost: ~$0.08 (1 target × 150 cap × Haiku rates)
  • Operator overrides via SF input for ad-hoc runs against different target models (e.g. `claude-sonnet-4-6` self-test)

IAM updates

  • `github-actions-lambda-deploy.json` — `alpha-engine-replay-concordance` added to LambdaUpdate + LambdaInvokeCanary lists. Asymmetric-IAM-grant antipattern compliance — 4th Lambda this shape; durable `lambda:CreateFunction` grant from data fix(iam): grant lambda:CreateFunction + PassRole(lambda) for first deploys #165 already covers create + update.
  • `deploy_step_function.sh` — SF role inline LambdaInvoke list updated with the new function ARN so SF can invoke it.

Test plan

  • 8 new SF wiring tests; suite 451 → 459
    • States present (CheckSkipReplayConcordance + ReplayConcordance)
    • SkipRationaleClustering reroute (now → CheckSkipReplayConcordance)
    • RationaleClustering success + Catch reroute (same)
    • SkipReplayConcordance: skip_replay_concordance flag → SaturdayHealthCheck
    • ReplayConcordance: live alias, payload required fields, 900s timeout, success + Catch routes, retry posture
  • Apply IAM via `apply.sh` on the live role (mostly a no-op since CreateFunction is already granted; the resource-list update is what's new)
  • Run `deploy_step_function.sh` to push the new SF definition + inline policy
  • First Sat SF run after backtester PR fix(backfill): filter universe writes by current constituents + escalate preflight #138 merges (and Lambda deploys via `bash infrastructure/deploy_concordance.sh`) fires the concordance state

🤖 Generated with Claude Code

Companion to alpha-engine-backtester PR #138 (concordance Lambda
implementation). Inserts the Saturday SF state that fires the
weekly cheap-model concordance pipeline.

SF chain after this PR:

  ... → EvalJudge{Weekly,FirstSaturday} → EvalRollingMean
      → CheckSkipRationaleClustering → RationaleClustering
      → CheckSkipReplayConcordance → ReplayConcordance
      → SaturdayHealthCheck → ...

Independent skip-gates per observability path:

- {"skip_eval_judge": true}        — bypass judge only
- {"skip_rationale_clustering":...} — bypass clustering only (was
  rerouted from SaturdayHealthCheck to CheckSkipReplayConcordance
  in this PR; the two are independent agent-justification signals)
- {"skip_replay_concordance": true} — bypass concordance only

Each Catch lands at SaturdayHealthCheck (eval observability —
failures must NOT halt the pipeline).

Default ReplayConcordance payload pins production cadence:

  - target_models: ["claude-haiku-4-5"] (Sonnet→Haiku concordance)
  - window_days: 56 (8 weeks trailing)
  - max_artifacts: 150 (fits 900s Lambda timeout at ~3-5 sec/call)

Operator overrides via SF input parameters for ad-hoc runs against
different target models (e.g. claude-sonnet-4-6 self-test).

IAM updates:

- github-actions-lambda-deploy.json: alpha-engine-replay-concordance
  added to LambdaUpdate + LambdaInvokeCanary (asymmetric-IAM-grant
  antipattern compliance — 4th time this shape, durable fix is the
  CreateFunction grant from PR #165 already covers create+update).
- deploy_step_function.sh: SF role inline LambdaInvoke list updated
  with the new function ARN so SF can invoke it.

Tests:

- TestStatesPresent: CheckSkipReplayConcordance + ReplayConcordance
  required.
- TestSkipRationaleClustering: rerouted assertion (now lands at
  CheckSkipReplayConcordance, not SaturdayHealthCheck).
- TestRationaleClustering: success + Catch reroutes (now CheckSkipReplay
  Concordance).
- TestSkipReplayConcordance: skip_replay_concordance flag → SaturdayHealthCheck.
- TestReplayConcordance: live alias, payload required fields
  (end_time_iso, target_models, window_days, max_artifacts), 900s
  timeout matches Lambda cap, success + Catch routes, retry posture.

Suite 451 → 459.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@cipher813 cipher813 merged commit 65df324 into main May 6, 2026
1 check passed
@cipher813 cipher813 deleted the feat/sf-replay-concordance-state branch May 6, 2026 02:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant