feat(data): delete ssm_secrets + .env + push-secrets.sh + spot .env sourcing (PR 9f)#241
Merged
Merged
Conversation
…env sourcing (PR 9f of .env deprecation) Closes L2998 PR 9f. Master-repo cleanup of the .env-deprecation arc: deletes the legacy ssm_secrets.py shim + its 2 production call sites (weekly_collector.py, lambda/handler.py) + the master .env file + infrastructure/push-secrets.sh + the .env-sourcing scaffolding in infrastructure/spot_data_weekly.sh + infrastructure/spot_drift_detection.sh (preamble + SCP-to-spot + inline source + ENV_SOURCE variable). Secrets now load exclusively from SSM via alpha_engine_lib.secrets.get_secret() per the PRs 1-8a substrate shipped 2026-05-12. Deploy ordering: merge AFTER alpha-engine-backtester PR 9c (whose spot scripts cross-reference this repo's .env until that PR merges). Plan doc: ~/Development/alpha-engine-docs/private/env-260514.md Observation gate cleared 2026-05-14 EOD via L320 + L323 verifications. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Discovered post-merge of PR 9e (research) — that agent caught the same issue in its repo's Dockerfile + Dockerfile.alerts. Both predictor + data have the same COPY ssm_secrets.py directive. Without this commit, the post-merge auto-deploy Docker build fails with COPY failed: file does not exist. Belongs in PR 9f's scope; the plan-doc Dockerfile audit only checked for .env mentions and missed this. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
cipher813
added a commit
that referenced
this pull request
May 16, 2026
…oval regression (#247) PR 9f (#241) removed `.env` sourcing from the spot bootstrap in favor of runtime get_secret() SSM lookups. That handled secrets, but the same `.env` was also the only thing exporting AWS_REGION — a plain env var (not a secret) that alpha_engine_lib.preflight.check_env_vars() hard- requires and boto3 needs as a default region. Result: 2026-05-16 Saturday SF DataPhase1 aborted ~1m52s in, at `weekly_collector.py --morning-enrich` preflight: RuntimeError: Pre-flight: required env vars missing: ['AWS_REGION'] Whole Saturday pipeline aborted (no downstream stale-data risk — it failed before any write). Fix: ENV_SOURCE (interpolated into every remote `run_remote bash` heredoc) now also exports AWS_REGION + AWS_DEFAULT_REGION from the dispatcher-side $AWS_REGION (already defaulted to us-east-1). Applied to both spot_data_weekly.sh and spot_drift_detection.sh — the identical #241 regression affects the Saturday DriftDetection state too. Regression test pins the ENV_SOURCE region exports in both scripts so a future ENV_SOURCE edit can't silently drop them again (shim-deletion launch-mechanism class). Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
cipher813
added a commit
that referenced
this pull request
May 16, 2026
…nv-var check (#248) Second facet of the #241/#242 .env-deprecation regression, surfaced by the 2026-05-16 Saturday SF recovery run: AWS_REGION fix (#247) let DataPhase1 clear preflight and MorningEnrich completed (polygon 913/921 + FRED 4/4 fetched fine via get_secret()), but `weekly_collector.py --phase 1` then aborted at preflight: RuntimeError: Pre-flight: required env vars missing: ['FRED_API_KEY', 'POLYGON_API_KEY'] Every collector AND both reachability probes in this file already resolve these keys via get_secret() (SSM). The only stale code was DataPreflight.run()'s `check_env_vars("FRED_API_KEY","POLYGON_API_KEY")` (and the phase2 FMP/FINNHUB/EDGAR equivalent) — an os.environ assertion the env-deprecation arc migrated every consumer away from but missed here. MorningEnrich slipped through because its preflight only checks AWS_REGION; phase1/phase2 hard-failed on the stale gate. Fix: AWS_REGION stays an env-var check (plain boto3 region, not a secret); the API keys now go through a new `_check_secrets()` helper that calls get_secret(required=False) — same <1s fail-fast intent, same RuntimeError shape, sourced from SSM (with get_secret's env fallback) instead of os.environ. phase2 had the identical latent bug and is fixed in the same change. Tests updated to the get_secret() reality (patch preflight.get_secret rather than os.environ); full suite 1050 passed, 1 skipped. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
PR 9f of the
.env-deprecation arc — master-repo cleanup. After this, alpha-engine-data has exactly one secret-resolution path (alpha_engine_lib.secrets.get_secret()→ SSM) and no.envartifacts anywhere in the repo or its spot dispatchers.Closes L2998 PR 9f. Plan doc:
~/Development/alpha-engine-docs/private/env-260514.md. Observation gate cleared 2026-05-14 EOD via ROADMAP L320 + L323 verifications.Changes
ssm_secrets.py(legacy bulk-load shim, md5c839b0bdc16b267c9c61e40dcfc4f6b7).load_secrets()import + call at the 2 production call sites:weekly_collector.py+lambda/handler.py(drops orphanedimport sysin handler too).infrastructure/push-secrets.shentirely (master.env-pushing tool — nothing to push anymore)..envat repo root (gitignored, local artifact)..env.exampleto a single-line SSM tombstone pointer..envsourcing frominfrastructure/spot_data_weekly.sh+infrastructure/spot_drift_detection.sh(preamble + SCP-to-spot + inlinesource /home/ec2-user/alpha-engine-data/.env+ theENV_SOURCEvariable assignment). PreservesXDG_CACHE_HOME+PYTHON_BIN/PYTHONPATHexports insideENV_SOURCE. Spot's IAM profilealpha-engine-executor-profilealready grantsssm:GetParameteron/alpha-engine/*, so Python entry points resolve secrets via SSM at startup.Diff stat: 7 files changed, 14 insertions(+), 396 deletions(-).
Deploy ordering
Must merge AFTER
alpha-engine-backtesterPR 9c. Backtester's spot scripts (spot_backtest.sh,spot_backtest_and_evaluate.sh) currently sourcealpha-engine-data/.envcross-repo. If this PR (9f) merges first, the next backtester spot launch would try to source a non-existent file and fail.Per the plan doc's sequencing: dashboard → lib → backtester → predictor → research → data → executor.
Test plan
1035 passed, 1 skipped, 5 warnings in 5.87s(1 skip + 5 FutureWarnings are pre-existing, unrelated).grep -rln 'from ssm_secrets\|import ssm_secrets\|load_secrets()' . --include='*.py' --include='*.sh' --exclude-dir='.venv' --exclude-dir='__pycache__'returns zero hits.bash -n infrastructure/spot_data_weekly.sh && bash -n infrastructure/spot_drift_detection.shexits cleanly..envmention audit on spot scripts: only comment-context historical references remain (the 2026-04-17 RAG_DATABASE_URL truncation comment, plus the new tombstone comments) — no active sourcing.ae-tradingwriteshealth/daily_data.jsonclean (next scheduled boot pulls latest main)..envSCP step.🤖 Generated with Claude Code