chore(eod-sf): remove redundant python -m builders.daily_append invocation#129
Merged
Merged
Conversation
…ation Follow-up to PR #128. The PostMarketData step in alpha-engine-eod-pipeline ran two python commands back-to-back: python weekly_collector.py --daily --only daily_closes python -m builders.daily_append The second one was a no-op duplicate. ``_run_daily`` runs daily_closes + feature_store + daily_append regardless of ``--only`` (the flag is only wired into ``_run_phase1``), so the second invocation just ran daily_append again — and as a bare CLI call it didn't get the new ``skip_if_exists`` flag PR #128 added, so on re-runs every ticker took the slow lib.write backfill path (904 × ~1.5s ≈ 22 min, over the SSM 1200s timeout — exactly what blew up the 2026-05-01 EOD SF rerun). Also drop the misleading ``--only daily_closes`` qualifier on the remaining invocation since it doesn't actually filter. Adds a regression test locking the SSM script to a single ``weekly_collector --daily`` invocation. Both infrastructure/step_function_eod.json and infrastructure/update_eod_pipeline_sf.sh updated; the deploy script needs to be run on ae-trading after merge to take effect. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
PostMarketDatastep randaily_appendtwice back-to-back: once viaweekly_collector --daily(which feat(daily_append): source-aware skip_if_exists for EOD re-runs #128 made source-aware) and again via a barepython -m builders.daily_appendCLI call (which still defaulted toskip_if_exists=False)._run_dailyalready runs daily_closes + feature_store + daily_append regardless of--only(the flag only filters in_run_phase1).Why now
The 2026-05-01 EOD SF rerun (after PR #126 merged + the universe was re-migrated) timed out at the SSM 1200s ceiling. With #128 deployed, the first invocation completes in seconds when today's row is already written. The second bare-CLI invocation still hits the slow
lib.writebackfill path on every ticker (~22 min) and re-introduces the timeout. Removing it drops total wall time below 3 min on re-runs and gets us back the 5/1 EOD reconcile run cleanly.Also dropped
The
--only daily_closesqualifier on the remaining invocation. It was misleading —--onlyis only wired into_run_phase1, not_run_daily, so the qualifier did nothing. The bare--dailyis the canonical invocation.Test plan
pytest tests/test_daily_append_skip_if_exists.py— 8/8 pass (newtest_eod_ssm_script_has_no_redundant_daily_appendregression locks the simplification).infrastructure/update_eod_pipeline_sf.shon ae-trading after merge to redeploy the SF.🤖 Generated with Claude Code