Skip to content

feat(sf): wire counterfactual rule fit Lambda into Saturday SF#168

Merged
cipher813 merged 1 commit into
mainfrom
feat/sf-counterfactual-state
May 6, 2026
Merged

feat(sf): wire counterfactual rule fit Lambda into Saturday SF#168
cipher813 merged 1 commit into
mainfrom
feat/sf-counterfactual-state

Conversation

@cipher813
Copy link
Copy Markdown
Owner

Summary

Companion to alpha-engine-backtester #140 (counterfactual Lambda implementation). Closes the agent-justification triple end-to-end — all three signals now fire from the Saturday SF on the same trailing-8-week corpus.

SF chain after this PR

```
... → EvalJudge{Weekly,FirstSaturday} → EvalRollingMean
→ CheckSkipRationaleClustering → RationaleClustering
→ CheckSkipReplayConcordance → ReplayConcordance
→ CheckSkipCounterfactual → Counterfactual
→ SaturdayHealthCheck → ...
```

Three independent skip-gates

Each lands on the next signal's gate rather than SaturdayHealthCheck — skipping one signal doesn't bundle-skip the others:

Flag Routes to
`skip_rationale_clustering: true` `CheckSkipReplayConcordance`
`skip_replay_concordance: true` `CheckSkipCounterfactual`
`skip_counterfactual: true` `SaturdayHealthCheck`

Default Counterfactual payload

```json
{
"end_time_iso.$": "$$.Execution.StartTime",
"window_days": 56,
"max_depth": 3
}
```

No `target_models` — counterfactual doesn't replay against a target model; sklearn fits a tree on actual (input → decision) pairs.

IAM updates

Test plan

  • 8 new SF wiring tests; suite 459 → 467
    • States present (CheckSkipCounterfactual + Counterfactual)
    • SkipReplayConcordance reroute (now → CheckSkipCounterfactual)
    • ReplayConcordance success + Catch reroute (same)
    • SkipCounterfactual: skip_counterfactual → SaturdayHealthCheck
    • Counterfactual: live alias, payload required fields, 600s timeout, success + Catch routes, retry posture
  • Apply IAM via `apply.sh` (resource-list update)
  • Run `deploy_step_function.sh` to push the new SF definition + inline policy
  • First Sat SF run after backtester feat(sf): PR 4c rolling-mean SF wiring + CloudWatch alarm #140 merges (and Lambda deploys via `bash infrastructure/deploy_counterfactual.sh`) fires the counterfactual state

Agent-justification dashboard surface

All five metrics emit under `AlphaEngine/Eval`:

Metric Source
`agent_quality_score` eval-judge
`agent_quality_score_4w_mean` eval-rolling-mean
`agent_rationale_template_concentration` clustering
`agent_cheap_model_concordance` concordance
`agent_counterfactual_rule_fit` counterfactual

🤖 Generated with Claude Code

Companion to alpha-engine-backtester #140 (counterfactual Lambda
implementation). Closes the agent-justification triple end-to-end —
all three signals now fire from the Saturday SF on the same trailing-
8-week corpus.

SF chain after this PR:

  ... → EvalJudge{Weekly,FirstSaturday} → EvalRollingMean
      → CheckSkipRationaleClustering → RationaleClustering
      → CheckSkipReplayConcordance → ReplayConcordance
      → CheckSkipCounterfactual → Counterfactual
      → SaturdayHealthCheck → ...

Three independent skip-gates, each landing on the next signal's gate
rather than the health check (so skipping one signal doesn't bundle-
skip the others):

- {"skip_rationale_clustering": true} → CheckSkipReplayConcordance
- {"skip_replay_concordance": true}    → CheckSkipCounterfactual
- {"skip_counterfactual": true}        → SaturdayHealthCheck

Default Counterfactual payload pins production cadence:

  - end_time_iso = SF execution start time
  - window_days = 56 (8 weeks)
  - max_depth = 3 ("3-deep rule")

No target_models payload — counterfactual doesn't replay against a
target model; sklearn fits a tree on actual (input → decision) pairs.

IAM updates:

- github-actions-lambda-deploy.json: alpha-engine-replay-counterfactual
  added to LambdaUpdate + LambdaInvokeCanary lists. Asymmetric-IAM-
  grant antipattern compliance — 5th Lambda this shape; durable
  CreateFunction grant from data #165 already covers create + update.
- deploy_step_function.sh: SF role inline LambdaInvoke list updated
  with the new function ARN so SF can invoke it.

Tests:

- TestStatesPresent: CheckSkipCounterfactual + Counterfactual added
  to required-states pin.
- TestSkipReplayConcordance: rerouted assertion (now lands at
  CheckSkipCounterfactual, not SaturdayHealthCheck).
- TestReplayConcordance: success + Catch reroutes (same).
- TestSkipCounterfactual: skip_counterfactual flag → SaturdayHealthCheck.
- TestCounterfactual: live alias, payload required fields
  (end_time_iso, window_days=56, max_depth=3), 600s timeout matches
  Lambda cap, success + Catch routes, retry posture.

Suite 459 → 467.

Composes with the agent-justification dashboard surface — all three
metrics emit under the AlphaEngine/Eval namespace:

  - agent_quality_score (eval-judge)
  - agent_quality_score_4w_mean (eval-rolling-mean)
  - agent_rationale_template_concentration (clustering)
  - agent_cheap_model_concordance (concordance)
  - agent_counterfactual_rule_fit (this one)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@cipher813 cipher813 merged commit 6a6df9e into main May 6, 2026
1 check passed
@cipher813 cipher813 deleted the feat/sf-counterfactual-state branch May 6, 2026 02:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant