Troubleshooting

Troubleshooting

Failure modes whose cause is visible in the code at HEAD. Each entry says what you see, why it happens, and what to change.

terraform init fails to acquire a state lock

The README's quick-start snippet writes dynamodb_table = "reportmate-terraform-locks", plural. bootstrap/main.tf creates <project_name>-terraform-lock, singular, and that is what backend.hcl.example uses. Take the value from terraform output lock_table in the bootstrap directory rather than from the README.

Browser gets a TLS error or a connection reset on HTTPS

The port 80 listener always redirects to https:// on port 443. The port 443 listener only speaks HTTPS when acm_certificate_arn is non-empty; otherwise it is configured as a plain HTTP listener on port 443. So with no certificate, any request to port 80 gets redirected into a TLS handshake against a listener that does not do TLS.

Set acm_certificate_arn to a certificate in the same region as the ALB. The HTTP-on-443 fallback exists only so a first apply can complete before a certificate is requested.

Dashboard loads but every API call goes to the wrong host

public_api_url is written into the frontend task definition as API_BASE_URL and NEXTAUTH_URL. It is a required variable with no default, and on a first apply you do not yet know the ALB DNS name. If it is still set to a placeholder, the server-rendered HTML will point the browser at that placeholder.

Read terraform output frontend_url, set public_api_url to the hostname users will actually type, and re-apply. The frontend task definition changes, which triggers a new deployment.

Sign-in fails or shows a configuration error

The dashboard defaults to the Entra ID provider and throws when AZURE_AD_CLIENT_ID, AZURE_AD_CLIENT_SECRET or AZURE_AD_TENANT_ID is missing; the frontend task definition sets none of them, and the dashboard has no Cognito provider. Add the settings for a supported provider to the frontend task definition in modules/containers. See step 8 of Getting-Started.

If you add a Cognito-backed provider, note that auth_callback_urls and auth_logout_urls default to http://localhost:3000/api/auth/callback/cognito and http://localhost:3000. Cognito rejects any callback not in that list.

Every /api/auth/session call returns 500

NextAuth raises MissingSecretError (NO_SECRET) without NEXTAUTH_SECRET. The module generates that secret and injects it from Secrets Manager, so this should not happen — unless the execution role cannot read the secret. Check that module.identity received module.auth.secret_arns; the inline policy on the execution role grants secretsmanager:GetSecretValue on exactly the ARNs it was passed and nothing else.

ECS tasks stop with ResourceInitializationError fetching secrets

The task execution role needs both secretsmanager:GetSecretValue and network reachability to the Secrets Manager endpoint. With enable_vpc_endpoints = false that means the NAT gateway must be healthy — the tasks are in private subnets with no public IP. If NAT is broken or you are in a region without the S3 gateway route working as expected, tasks will fail before your container ever starts. Setting enable_vpc_endpoints = true removes the NAT dependency for ECR, CloudWatch Logs, and Secrets Manager at a fixed hourly cost.

API task crash-loops shortly after starting

The API's container health check runs curl -f http://localhost:8000/api/v1/health, and that endpoint opens a database connection. Under database memory pressure or write contention a fresh connection can take longer than the check allows. The task definition already compensates with a 15s timeout, 5 retries, and a 90s start period. If it still loops, the database is the problem, not the check — look at db_instance_class and db_allocated_storage before raising the thresholds.

Target group reports no healthy targets right after apply

Expected on a first apply. The ECR repositories are created empty and both task definitions reference <repo-url>:<tag> with tag defaulting to latest. The services cannot pull an image that does not exist. Push images, then force a new deployment.

scripts/*.sh target a cluster that does not exist

The scripts were extracted from a monorepo and carry hardcoded names that do not all match what Terraform creates:

Script	Hardcoded	Actually created
deploy-api.sh	CLUSTER_NAME="reportmate-prod"	<project>-<env>-cluster
status.sh	CLUSTER_NAME="reportmate-prod"	<project>-<env>-cluster
deploy-containers.sh	reportmate-prod-cluster	correct only if project_name = reportmate and environment = prod
deploy-demo-loop.sh	reportmate-prod-cluster	same

deploy-containers.sh also builds the frontend from $MONO_ROOT/apps/www, and deploy-demo-loop.sh builds from scripts/Dockerfile.demo-loop, both two directories above the repo root — paths that only exist in the monorepo. Use the scripts as a reference for the AWS CLI invocations rather than running them unmodified.

The GitHub Actions workflow never runs, or fails immediately

.github/workflows/terraform.yml filters on paths: ['infrastructure/aws/**'] and sets TF_DIR: infrastructure/aws. Neither exists in this repo. Fix both before relying on CI, and add TF_VAR_public_api_url to the plan and apply steps — it is required and the workflow does not currently pass it.

terraform destroy refuses to delete the database

aws_db_instance.main has deletion_protection = true and a prevent_destroy lifecycle block. Both must be removed in code, and the removal applied, before a destroy will proceed. It also takes a final snapshot named <prefix>-final-snapshot; if a snapshot with that name already exists from a previous teardown, the destroy fails on the name collision.

The bootstrap state bucket has prevent_destroy for the same reason.

No budget alerts arrive

aws_budgets_budget.daily is created only when alert_email is non-empty — the whole budget, not just the email subscription. The NAT egress alarms are independent and are created whenever nat_bytes_out_alarm_gb > 0, but with no email subscription on the SNS topic they publish into a topic nobody reads.

Second deployment in the same account collides

Three name collisions to expect:

• The Cognito hosted-UI domain is <project>-<env> and is globally unique across all AWS accounts.
• ECR repository names use project_name only, with no environment, so two environments in one account share repositories.
• <project>-<env>-storage and <project>-terraform-state are globally unique S3 bucket names.

Change project_name for the second deployment.

Things that look configurable but are not

• domain_name and enable_custom_domain are declared and referenced by nothing. There is no CloudFront, ACM, or Route 53 resource in the repo.
• The networking module's vpc_endpoint_security_group_id variable is unused; the endpoint security group allows 443 from the whole VPC CIDR instead.
• The containers module's database_security_group_id variable is unused; the database module's own ingress rule is what opens 5432.
• The identity module's ecr_repository_arns variable is unused. The root module passes ECR repository URLs into it, not ARNs — harmless only because nothing reads it.
• Autoscaling bounds (1–5 tasks, 70% CPU) and the maintenance schedule (02:00 UTC) are hardcoded in their submodules.
• The SQS ingest queue is created and exported but not connected to any container by this module.