fix(sf-daily): set -eo pipefail on every SSM RunShellScript block#209
Merged
Conversation
Without `pipefail`, a non-zero exit from `python ...` upstream of a pipe to `tee` is silently masked by tee's exit 0. SSM RunShellScript then reports `ResponseCode: 0, Status: Success` and the SF moves on with no indication that the step actually failed. This was the load-bearing cause of the 2026-05-11 silent-MorningEnrich cascade: `python weekly_collector.py --morning-enrich 2>&1 | tee -a /var/log/morning-enrich.log` exited 1 from the constituents-preflight raise, the SF reported MorningEnrich Success, PredictorInference + RunMorningPlanner ran against stale daily_data, and the planner aborted minutes later with "daily_data: 46h stale". Add `set -eo pipefail` as the first command of every SSM RunShellScript block in step_function_daily.json: - CheckTradingDay - MorningEnrich (pipe to tee — load-bearing) - RunMorningPlanner (pipe to tee — load-bearing) - RunDaemon New wiring test (`tests/test_sf_ssm_pipefail_wiring.py`) parametrizes over all three SF defns (saturday + weekday + eod) and asserts every SSM RunShellScript command array begins with a `set ... pipefail` line. Accepts both `set -eo pipefail` (Saturday + weekday) and `set -o pipefail` (EOD) conventions — the absence of `pipefail` is the bug being prevented. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
set -eo pipefailas the first command of every SSMRunShellScriptblock ininfrastructure/step_function_daily.json:CheckTradingDay,MorningEnrich,RunMorningPlanner,RunDaemon.tests/test_sf_ssm_pipefail_wiring.pyparametrizes over all three SF defns (Saturday + weekday + EOD) and asserts every SSMcommandsarray begins with aset ... pipefailline. Catches future regressions where a state is added or rewritten without the flag.Why
Without
pipefail,python ... 2>&1 | tee -a /var/log/foo.logsilently masks any non-zero exit frompythonbecauseteereturns 0. SSMRunShellScriptthen reportsResponseCode: 0, Status: Successand the SF moves past the failure.This was the load-bearing cause of the 2026-05-11 silent-MorningEnrich cascade:
python weekly_collector.py --morning-enrichexited 1 (constituents-preflight raise)| teereturned 0 → SSM Success → SF advanced through PredictorInference + RunMorningPlanner against stale ArcticDB / S3 daily_datadaily_data: 46h stale→ no order book written → daemon Telegram alertAdding
set -eo pipefailmakes any future MorningEnrich / planner failure propagate to SSMResponseCode != 0, which the SF's existingCatch[States.ALL]already routes toHandleFailure/MorningEnrichFailed. Saturday SF already uses this convention in every state (8 instances); this PR brings the daily SF in line.Test plan
pytest tests/test_sf_ssm_pipefail_wiring.py— 3/3 pass (parametrized over saturday/weekday/eod SF defns)pytest tests/— 714 passed, 1 skippedpython -c "import json; json.load(open('infrastructure/step_function_daily.json'))"start-execution) will exercise the new flag end-to-end.Independent of #207, #208
This PR touches only
infrastructure/step_function_daily.json+ a new test file. No conflict with #207 (constituents.py) or #208 (constituents.py + .gitignore). All three can land in any order.🤖 Generated with Claude Code