Restore alpha-engine-data in boot-pull REPOS (pair with spot migration)#20
Merged
Merged
Conversation
Reverts the removal from #19. The Saturday Step Function still runs RAGIngestion + DriftDetection on the micro from alpha-engine-data, and the new DataPhase1 spot launcher (cipher813/alpha-engine-data#44) lives at alpha-engine-data/infrastructure/spot_data_phase1.sh — the micro invokes it as a dispatcher and needs the repo checked out. Context: #19 assumed ae-dashboard had no runtime need for alpha-engine-data once health_checker + trading_calendar moved here. That was wrong — the Saturday SF has 4 separate steps that target the micro from alpha-engine-data (DataPhase1, RAGIngestion, DriftDetection, SaturdayHealthCheck). I missed this when recommending #19. After cipher813/alpha-engine-data#44 merges and RAG/Drift also migrate to spot (planned follow-up), alpha-engine-data may still need to be cloned on the micro for the launcher scripts but the heavy .venv becomes unnecessary. At that point this line can be removed again along with a lean-clone pattern. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Reverts the REPOS trim from #19. Pair with cipher813/alpha-engine-data#44 (DataPhase1 spot migration).
Why
My earlier audit missed that the Saturday Step Function targets ae-dashboard for four steps that run code from alpha-engine-data:
So alpha-engine-data needs to stay checked out on ae-dashboard. Removing it in #19 was premature — that audit only looked at crontab + dashboard-repo imports and didn't check the Saturday SF.
Change
```diff
REPOS=(
/home/ec2-user/alpha-engine-config
/home/ec2-user/alpha-engine-research
/home/ec2-user/alpha-engine-dashboard
/home/ec2-user/flow-doctor
)
```
After this merges — manual deploy steps
```
ae-dashboard: re-clone the repo (rm -rf was run earlier)
ae-dashboard "git clone git@github.com:cipher813/alpha-engine-data.git /home/ec2-user/alpha-engine-data"
rebuild .venv (heavy — yfinance, polygon, FRED, ArcticDB, SEC EDGAR tools)
ae-dashboard "cd /home/ec2-user/alpha-engine-data && python3 -m venv .venv && source .venv/bin/activate && pip install -r requirements.txt"
```
After that, the next `boot-pull.timer` fire at 12:00 UTC will keep it refreshed going forward.
Follow-up
Once RAGIngestion + DriftDetection also migrate to spot (planned next week), the heavy `.venv` on the micro can go away. The repo itself will still need to be checked out for the launcher scripts, but the memory footprint on the micro shrinks significantly.
🤖 Generated with Claude Code