fix(sf): stamp run_date once at InitializeInput, thread to spot stages#253
Merged
Conversation
The 2026-05-17 Saturday SF failed at the Evaluator: backtest/{date}/
artifacts were keyed off a per-stage wall-clock date. Backtester /
Parity / Evaluator are each a separate spot instance running its own
spot_backtest.sh; a ~2.5h run that straddled UTC midnight split the
prefix (Backtester wrote backtest/2026-05-17/, Evaluator's post-
midnight spot looked in backtest/2026-05-18/ → "All critical
simulation artifacts missing").
Producer side of the fix (consumer side = alpha-engine-backtester
spot_backtest.sh honoring injected RUN_DATE + pinning the backtest
stage to it):
1. InitializeInput now stamps run_date = date($$.Execution.StartTime)
via States.Format/StringSplit/ArrayGetItem, merged so a manually
passed run_date still wins (same user-input-wins semantics as the
existing sns_topic_arn default). StartTime is constant for the whole
execution → every stage gets ONE date.
2. Backtester / Parity / Evaluator commands rebuilt via States.Array to
inject `export RUN_DATE='<$.run_date>'` immediately before the
`bash infrastructure/spot_backtest.sh` launch. spot_backtest.sh
resolves RUN_DATE from env (wall-clock fallback for ad-hoc runs).
run_date propagates across the Research/Predictor Parallel safely:
ResultPath=$.parallel_result (sub-key, root preserved), same as the
existing $.ec2_instance_id which Backtester already consumes post-join.
Validated: `aws stepfunctions validate-state-machine-definition` → OK,
zero diagnostics.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The commands→commands.$ States.Array conversion (run_date injection) broke 7 wiring tests that assert on the static `commands` key. The assertions' intent (pipefail-first, trap-before-work, skip-stages flags, evaluator.log tee, stage isolation) is unchanged — they just needed to read through the intrinsic. - tests/sf_command_utils.py: shared extract_commands() that returns the ordered command strings whether stored as `commands` (list) or `commands.$` (States.Array intrinsic), unescaping ASL literals and rendering a States.Format element as its template. - test_sf_evaluator_wiring.py / test_sf_backtest_parity_split_wiring.py: route through extract_commands (7 tests fixed). - test_sf_run_date_threading.py: NEW — locks InitializeInput stamping run_date from $$.Execution.StartTime, user-input-wins ordering, and `export RUN_DATE` injected before the spot launch in all 3 stages. Full suite: 1219 passed, 1 skipped, 0 failed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
9 tasks
cipher813
added a commit
that referenced
this pull request
May 22, 2026
…m_log_capture CLI (#290) Closes the 2026-05-22 Friday-PM dry-pass failure (Alpha Engine Saturday Pipeline FAILED at MorningEnrich, exit 127, "trap: s3: invalid signal specification"). Lifts the inline `trap 'aws s3 cp /var/log/X.log "s3://..." || true' EXIT` + `tee /var/log/X.log` pattern out of JSON and into the alpha_engine_lib.ssm_log_capture Python CLI (lib v0.25.0). **Root cause** of the dry-pass break: PR #253 (merged 2026-05-17) flipped 8 Saturday-SF spot states from plain `commands` JSON arrays to `commands.$ States.Array(...)` so they could splice `$.run_date` / `$.preflight_args` via States.Format. Inside States.Array arg strings, ASL's documented escape for an inner single quote is `\'` — but the AWS ASL evaluator does NOT unescape `\'` to `'` in practice; it passes the backslash through literally. The trap line `'trap \'cmd\' EXIT'` therefore rendered into the SSM `_script.sh` as `trap \'cmd\' EXIT`. Bash interpreted the `\'` outside quotes as a literal apostrophe stripped of its quoting power, then word-split the line and passed every token after `aws` to `trap` as a signal name. Exit 127, line 7. Same defect lurked in ALL 8 Saturday-SF spot states (MorningEnrich first only because it's the first to run; everything downstream would have failed the same way). The 2026-05-22 Friday-PM shell-run dry-pass — designed exactly to catch break classes like this ~12h before the Saturday 09:00 UTC firing — exposed the latent regression on its first execution under the broken pattern (no Saturday SF had run between #253 merge and the dry-pass). The Friday-PM dry-pass safety net worked as designed. **Fix shape (institutional, per CLAUDE.md SOTA sub-sub-rule):** Each affected state now invokes a single States.Format-rendered token list with no bash trap and no inner single quotes: /home/ec2-user/alpha-engine-dashboard/.venv/bin/python \ -m alpha_engine_lib.ssm_log_capture run \ --slug X --log /var/log/X.log -- bash <launcher> ... The lib CLI internalizes the tee + S3 ship-on-exit + exit-code propagation. No ASL escape surface. Tested independently in lib v0.25.0 (20 unit tests; see alpha-engine-lib PR #57). States converted (slug → launcher): - morning-enrich → spot_data_weekly.sh --morning-enrich-only - data-weekly → spot_data_weekly.sh --phase1-only - rag-ingestion → spot_data_weekly.sh --rag-only - predictor-training → spot_train.sh --full-only (alpha-engine-predictor) - drift-detection → spot_drift_detection.sh - backtester → spot_backtest.sh --skip-stages=parity,evaluator (alpha-engine-backtester) - parity → spot_backtest.sh --skip-stages=backtest,evaluator (alpha-engine-backtester) - evaluator → spot_backtest.sh --skip-stages=backtest,parity (alpha-engine-backtester) The `States.Format('export RUN_DATE=\\'{}\\'', $.run_date)` line in Backtester/Parity/Evaluator stays as-is — `States.Format` template unescaping IS reliable (different ASL code path than States.Array arg unescaping). **Test gaps closed in the same diff:** 1. `test_sf_ssm_pipefail_wiring._iter_ssm_command_blocks` previously only iterated plain `commands` arrays — silently skipped every `commands.$ States.Array(...)` state. That was why the broken trap form went undetected from #253 (2026-05-17) until the dry-pass actually fired in production (2026-05-22). The helper now uses the shared `tests.sf_command_utils.extract_commands` to handle both forms uniformly. 2. `test_long_ssm_steps_ship_log_to_s3` now accepts EITHER an inline bash trap (plain `commands` form, used by weekday + EOD SFs) OR the alpha_engine_lib.ssm_log_capture CLI (commands.$ States.Array form, used by Saturday SF). Either shape satisfies the S3-log-capture invariant; the test enforces presence + correctness without prescribing one form. 3. `tests/fixtures/sf_prekeystone_spot_commands.json` regenerated from the new SF (the fixture's docstring explicitly permits this for "deliberate, reviewed change to a spot state's absent-path command" — this is exactly that case). Pre-keystone form had the broken `\'`-escape trap baked in; new baseline reflects the lib-CLI shape. The Friday-PM keystone byte-identicality proof still holds (absent path = lib-CLI invocation with preflight_args=""). 4. `test_morning_enrich_has_s3_log_trap_before_work` and the analogous parity test renamed/rewritten to assert the lib-CLI presence instead of the inline-trap presence. Both also assert that NO inline trap coexists with the lib CLI in the same state (would race on /var/log). 5. `test_spot_command_carries_preflight_only_under_shell_run` updated to assert the new shell_run command shape (lib CLI + `--` separator + launcher + --preflight-only). **Dependency chain (must merge in order):** 1. alpha-engine-lib #57 — adds ssm_log_capture, auto-tags v0.25.0 2. alpha-engine-data #289 — bumps lib pin to v0.25.0 (spot side) 3. alpha-engine-predictor #188 — bumps lib pin (PredictorTraining spot) 4. alpha-engine-backtester #243 — bumps lib pin (Backtester/Parity/Evaluator spots) 5. alpha-engine-dashboard #118 — bumps lib pin (ae-dashboard = SSM target, hosts the .venv where the lib CLI runs) 6. **THIS PR** — converts SF JSON to invoke the lib CLI 7. Run alpha-engine-data/infrastructure/deploy_step_function.sh 8. Redrive the Friday-PM dry-pass; confirm 8/8 spot states green 9. Saturday 09:00 UTC SF runs cleanly Suite: 1430 passed, 1 skipped, 7 warnings (FutureWarnings unrelated to this change). Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
7 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Producer side (consumer = alpha-engine-backtester#223). InitializeInput stamps
run_date = date($$.Execution.StartTime)(manual run_date still wins); Backtester/Parity/Evaluator commands rebuilt viaStates.Arraytoexport RUN_DATEbefore the spot launch. Propagates safely across the Parallel (ResultPath=$.parallel_result, root preserved — same as $.ec2_instance_id).validate-state-machine-definition→ OK, 0 diagnostics. Deploy: requires applying the SF def (see deploy note in thread) — not live on merge alone unless deploy-infrastructure auto-runs.🤖 Generated with Claude Code