Skip to content

Deployment issues

Jump to bottom
Ram Vallamsetti edited this page Oct 14, 2026 · 4 revisions

Deployment issues

Issues hit during Terraform apply, GitHub Actions Deploy, Cognito/site cutover, and ops for ram290476/investor-dashboard, and how they were fixed. Companion page: Build issues.

Account / region context: AWS 308639168050, primary us-west-1, deploy role invdash-terraform-deploy, site CloudFront (e.g. https://dbkmugvl0zmt0.cloudfront.net).

OIDC assume-role failed after first Deploy wiring
Symptom Deploy failed at configure-aws-credentials with Not authorized to perform sts:AssumeRoleWithWebIdentity. Site still showed Access Denied until this worked.
Cause GitHub’s OIDC sub for this repo uses immutable IDs (repo:owner@id/repo@id:environment:production), not the legacy repo:owner/name:environment:production form alone. Trust policy only matched one shape.
Remediation PR #13 updated the role trust to accept both sub formats, plus exact repository_id / repository_owner_id / ref / job_workflow_ref. Compare CloudTrail’s failed AssumeRoleWithWebIdentity userIdentity.userName (sub) to the table in docs/operations.md. Trust changes need an administrator apply (CI cannot PutRolePolicy on itself).

Empty jobs_image_uri apply destroyed the jobs stack
Symptom Deploy run after PR #14 (e.g. 37496345092) failed mid-run but destroyed ~95 job resources (Lambdas, schedules, IAM). Lake/site remained; collectors gone.
Cause Workflow “bootstrap” step ran terraform apply -var='jobs_image_uri='. With jobs already in state, empty URI → enabled_jobs = {} → Terraform plans full job destroys.
Remediation 1) Admin restore apply with the last good ECR tag (e.g. …/invdash/jobs:<sha>-1), never an empty URI. 2) PR #15: skip empty-image apply once jobs/ECR exist; refuse apply if JOBS_IMAGE_URI unset; fix SIGPIPE bootstrap check (Build issues). 3) Local admin plans in a worktree must pass the deployed jobs_image_uri and sec_user_agent or they will plan the same 95 destroys — never apply that plan.

CI denied PutRolePolicy on the deploy role (PassRole / Cognito)
Symptom Deploy apply AccessDenied on iam:PutRolePolicy for invdash-terraform-deploy when expanding PassRole (e.g. Cognito SMS MFA role).
Cause By design: explicit deny of self-modification on the deploy role / OIDC / boundary so CI cannot privilege-escalate.
Remediation Apply module.github_deploy changes (trust, iam-for-this-stack PassRole allowlist, boundary) with AdministratorAccess (SSO profile invdash-admin) before expecting CI to succeed. Do not weaken the self-modification deny.

Cognito SMS MFA: PassRole / registration blockers
Symptom Enabling SMS MFA failed with Cognito AccessDeniedException (PassRole to invdash-cognito-sms). Separately, US SMS needs toll-free registration (can take many business days).
Cause Deploy role lacked PassRole to Cognito until admin allowlist update; SMS sandbox + registration is outside Terraform.
Remediation Short term: Cognito MFA off (password-only), 15‑minute auth session (AuthSessionValidity), remove SMS role — applied intentionally after user approval; PR #15 encodes MFA-off in Terraform. Identity Center / console MFA for AWS remains required. SMS registration forms were prepared report-only and not submitted unless explicitly asked.

Cognito TOTP “invalid or expired” codes
Symptom Authenticator codes rejected during MFA setup; login flaky.
Cause Default Cognito auth session window was 3 minutes; stale QR / old authenticator secrets from repeated setup attempts.
Remediation Raised client auth_session_validity to 15 minutes. Prefer a fresh authenticator entry after any reset. MFA later disabled for the app (see above); AWS SSO MFA unchanged.

Config delivery blocked by audit bucket Object Lock
Symptom AWS Config could not deliver to the audit bucket.
Cause Audit bucket Object Lock / retention blocked Config’s delivery prefix.
Remediation PR #10: dedicated Config bucket (separate from Object Locked audit storage). Re-verify Config recorder/delivery channel after apply.

First stack apply: PowerUserAccess insufficient
Symptom Plan/apply under PowerUserAccess blocked IAM roles, password policy, Access Analyzer, etc.
Cause Managed PowerUserAccess excludes IAM (and related) that the stack creates on day one.
Remediation First full apply used SSO AdministratorAccess (rvellamsetti). Later routine deploys use OIDC invdash-terraform-deploy (PowerUser + bounded IAM). Keep admin only for bootstrap and github_deploy changes.

Region / DR defaults vs repo defaults
Symptom Early plans assumed us-east-1; product decision was primary us-west-1 with us-west-2 DR and us-east-1 only where required (CloudFront / IAM sign-in alarms).
Cause Repo defaults differed from the chosen deployment region.
Remediation Infra PR switched defaults; tfvars / GitHub AWS_REGION / TERRAFORM_TFVARS must stay aligned with us-west-1.

Site live but dashboard charts empty
Symptom Cognito password login works; UI shows no data.
Cause Data lake empty (serving/dashboard.json missing). Yahoo backfill never invoked after job restore; all 14 SSM API keys still REPLACE_ME. Schedules alone do not start the first ticker backfill.
Remediation Invoke invdash-backfill once with the ticker payload (no API keys required for Yahoo). Then rotate real keys with rotate-key.sh — see Data collectors API keys. Optional smoke: invdash-daily-prices after Alpaca keys are set.

Alert email / Cognito username details
Symptom Ops/security mail went to an old address; Cognito username is case-sensitive.
Cause Alert emails updated to ram.Vellamsetti@gmail.com; Cognito user created as ram.Vellamsetti@live.com (exact casing).
Remediation Keep tfvars alert list current; sign in with the exact Cognito username string. Confirm SNS topic subscriptions if alerts are missing.

Oct 13, 2026: CloudWatch alarm fixes (#178–#181) and post-deploy steps
Symptom CloudWatch alarms firing: site 5xx, prefs-api SERVING_READ_FAILED, Lambda errors/throttles, failed job runs.
Cause Account Lambda concurrency cap of 10, prefs-api missing s3:ListBucket, FINRA 204 empty bodies, changed release-day sources, LeaseHeld errors (details below).
Remediation PRs #178–#181 merged and deployed; Lambda quota increase pending; post-deploy checklist in progress.

1. First step: AWS sign-in approval

  • The box's admin profile invdash-admin uses IAM Identity Center SSO device login: aws sso login --profile invdash-admin --use-device-code --no-browser.
  • Ram must approve the device code at https://d-91677448af.awsapps.com/start/#/device. Codes expire in about 10 minutes.
  • Codes CNBM-NKHV and NDQV-DHXT expired unapproved; NVMW-WHSK was approved at about 7:24 PM PT. No verification ran before approval.

2. Root causes (diagnosis Oct 13)

  • Account Lambda concurrency cap of 10 with no reserved concurrency: TickerAdded fans out to 5 jobs, plus Job Finished → status-feed/dashboard-build, plus prefs traffic. This throttled prefs-api and caused the site 5xx.
  • prefs-api missing s3:ListBucket on serving/stock/*, so a missing object returned 403 and SERVING_READ_FAILED.
  • FINRA returning 204 with an empty body, which was parsed as JSON.
  • Release-day sources changed: BLS cpi.ics 404, Cleveland Fed layout change, Census 400 from the query.
  • LeaseHeld raising errors and retries.
  • Ruled out: company_ir, the 8-K 404, KMS.

3. Fixes

  • #178: prefs-api s3:ListBucket on serving/stock/*; a missing ticker returns 404 STOCK_NOT_READY; optional prefs_api_reserved_concurrency var (set to 3 after the quota increase).
  • #179: short-interest treats 204/empty as no rows, with retry/backoff for non-JSON.
  • #180: LeaseHeld logs job_skipped and exits cleanly via the shared job_handler.
  • #181: BLS bls.ics feed, Cleveland parser for the transposed tables, Census cell_value/BOPG.
  • Ram approved the merges at 6:17 PM PT; merged about 6:18 PM PT. Deploys finished 6:25 (#178), 6:33 (#179), 6:39 (#180) and 6:44 PM PT (#181).
  • Note: the queued merges cancelled CI runs, which were re-run one by one.

4. Quota

  • AWS Lambda concurrent executions in us-west-1: 10 → 1000. Request 615df9a25e17402e89de1fd03c366355pjT7TyR9 filed at 10:51 AM PT Oct 13; status: approved (CASE_CLOSED at 12:23 PM PT; account concurrency now 1000).

5. Post-deploy checklist (completed Oct 13)

  1. Verify the deploy: Lambdas updated, ListBucket present, #180 code live.
    Results (7:24–7:27 PM PT): verified. Five job Lambdas run image 2449022 (#181 merge, includes #178–#181); prefs-api updated 6:25 PM PT with STOCK_NOT_READY; role has ListBucket on catalog/* and serving/stock/*; #180 skip_when_leased=True confirmed in the deployed image.
  2. Purge invdash-job-dlq (112 stale Oct 7–12 messages, mostly dashboard-build LeaseHeld plus already-fixed bugs) ONLY after #180 is verified, and only with Ram's explicit go-ahead.
    Results: 118 messages purged at 7:25:57 PM PT; queue at 0 by 7:27 PM PT.
  3. Rerun invdash-short-interest and invdash-news-sentiment (#177 all-ticker news check).
    Results: short-interest returned 200 with no new FINRA dates (empty reply handled). news-sentiment returned 200, 102 rows; all 12 watchlist tickers have RSS headlines. Finnhub has returned empty lists since 6:16 PM PT (suspected date-window issue, unconfirmed).
  4. Check alarms (invdash-lambda-errors, failed-runs, throttles, site-api-5xx, prefs-api-serving-read-failed) and the quota status.
    Results: all alarms OK except dlq-not-empty (pending re-evaluation). lambda-errors at 3:46 PM PT came from short-interest (pre-#179); 5:41 PM and 6:41 PM PT came from invdash-regulatory-feeds (polars ComputeError on name strings), which is still failing hourly and is NOT fixed by these PRs. Quota: CASE_CLOSED at 12:23 PM PT; account concurrency is now 1000. Reserved concurrency for prefs-api is not set yet.

Hard rules (do not repeat)

  1. Never apply a plan that destroys the jobs stack because jobs_image_uri is empty or wrong.
  2. Never put provider API keys in Git, wiki, chat, or TERRAFORM_TFVARS — only rotate-key.sh → SSM.
  3. Never let CI modify invdash-terraform-deploy / OIDC / boundary; use admin apply.
  4. Protect GitHub environment production with required reviewers and main only.

Related docs

  • docs/operations.md — deploy role trust table, empty-image rules, Cognito MFA note
  • Build issues — CI / workflow / for_each / SIGPIPE
  • Data collectors API keys — key signup and install

Clone this wiki locally