Troubleshooting

Troubleshooting

Failure modes whose cause is visible in the module code. Each entry names the symptom, the reason, and the fix.

Plan and apply failures

"API pool demand … exceeds usable Postgres connections"

API pool demand (api_db_pool_max * api_max_replicas) exceeds usable Postgres
connections (db_max_connections - db_connection_reserve). Lower the pool or
replica count, or raise max_connections on the server.

This is the precondition on the API container app. Every replica opens its own pool, so a fully scaled-out API must fit inside the server's max_connections with a reserve left for superuser, autovacuum and migrations.

Either lower api_db_pool_max or api_max_replicas, or raise db_max_connections — in its own apply, because max_connections is a static Postgres parameter and changing it restarts the server. db_connection_reserve is a containers module variable (default 15) and is not exposed at the root.

The Function App fails to create with the default SKU

function_app_sku defaults to "Y1" at the root, but the functions module builds an azurerm_function_app_flex_consumption, which requires the Flex Consumption tier. Y1 passes the module's own validation — it is in the allowed list — and is then rejected by Azure at create time.

Set function_app_sku = "FC1", which is what terraform.tfvars.example uses.

terraform init fails asking for backend configuration

versions.tf declares backend "azurerm" {} with no arguments. Supply them:

terraform init -backend-config=backend.tfvars

See State-and-Backends.

Validation errors

Message	Cause
Azure allows a retention window between 7 and 35 days.	db_backup_retention_days outside 7–35
Environment must be 'dev', 'prod', or 'both'.	environment set to something else, such as staging
Sign in audience must be one of: …	auth_sign_in_audience typo
Auth providers must be one of: azure-ad, google, credentials.	Unknown entry in auth_providers
log_daily_quota_gb must be -1 (uncapped) or at least 0.1 GB.	A quota between 0 and 0.1
SKU must be a valid App Service Plan tier	function_app_sku not in the module's list

The terraform.tfvars.example file shows environment = "staging" in a commented block. That value does not pass validation.

Name collisions on the first apply

Three names must be globally unique and the module does not add a suffix to them:

• storage_account_name — the root passes use_exact_name = true, so the random_id suffix in the storage module is bypassed entirely.
• key_vault_name — 3–24 letters, digits and hyphens, and a soft-deleted vault of the same name blocks reuse for the retention period.
• function_app_name — the Function App's own storage account is derived from it, so a long name can also produce an invalid storage account name.

postgres_server_name is the exception: leave it empty and the module generates reportmate-database-<random hex>.

Role assignment failures

The identity, containers and key_vault modules all create azurerm_role_assignment resources. Contributor is not enough — the apply principal needs User Access Administrator or an equivalent role on the scope. For function_app_external_secret_vault_ids, that scope is a vault in someone else's resource group, and owning this one does not imply it.

Runtime failures

Devices stop checking in, but the dashboard looks fine

The API container app's ingress must be external. If it is set to internal, the dashboard keeps working over the Container Apps environment's internal DNS while every agent gets a 404 — and nothing surfaces the problem until someone notices devices going stale. The module carries an explicit warning comment above the ingress block for exactly this reason.

az containerapp show --name reportmate-functions-api --resource-group rg-reportmate-prod --query properties.configuration.ingress.external

That must be true.

Every apply rolls the running container image back

Both container apps put template[0].container[0].image in ignore_changes because the deploy pipeline owns the running tag. If someone removes those ignore_changes entries, each apply reverts the image to whatever frontend_image_tag / api_image_tag say in tfvars (latest by default).

Symptom: a deploy succeeds, then the next unrelated terraform apply silently downgrades production.

Everyone is signed out after every deploy

Two Terraform resources writing the same nextauth-secret vault entry cause this: each apply finds the other's value as the latest version and writes its own back, rotating NEXTAUTH_SECRET and invalidating every session. The removed block at the end of main.tf drops the auth module's copy from state without deleting the vault secret, leaving key_vault as the single writer. Do not re-add a second one.

The app registration has no identifier URI

identifier_uris on both azuread_application resources is Optional but not Computed. With no value in config, the provider reads the URI back from Entra, sees that config wants none, and strips it on every apply — racing the provisioner that sets it. Both applications therefore put identifier_uris in ignore_changes, leaving a null_resource running az ad app update as the single writer.

That provisioner uses on_failure = continue, so a transient "Insufficient privileges" during Entra replication is swallowed. It re-runs on every apply because of the always_run = timestamp() trigger, so the fix for an empty URI is usually just: apply again, from a host with a working az login.

The Function App cannot read Key Vault secrets

The functions module grants access to the module's own vault with azurerm_key_vault_access_policy. That vault is created with rbac_authorization_enabled = true, and access policies are inert on an RBAC-authorized vault. If a function reads a secret and gets nothing, grant the Function App's identity Key Vault Secrets User on the vault instead, and declare that grant in Terraform rather than clicking it in — the apply will not recreate a manual grant, and nothing will tell you it is missing.

The external_secret_vault_ids path already does it correctly, with an RBAC role assignment.

A Key Vault reference resolves to an empty string

An app setting of the form @Microsoft.KeyVault(SecretUri=...) in function_app_additional_settings resolves at runtime, not at apply time. If the identity cannot read the referenced secret, the setting is empty and the function fails with a confusing downstream error rather than an auth error.

An app setting you added by hand disappeared

Terraform owns the Function App's app_settings outright. A setting added with az functionapp config appsettings set survives until the next apply and then vanishes. Put it in function_app_additional_settings.

Ingestion stopped part-way through a day

log_daily_quota_gb (default 2) pauses Log Analytics ingestion for the remainder of the UTC day once reached, and app_insights_daily_cap (default 1) does the same for Application Insights. Both are runaway backstops. If telemetry goes quiet at a consistent time each day and resumes at midnight UTC, you are hitting a cap, not losing data at the source.

The database went read-only

PostgreSQL Flexible Server locks read-only when storage fills. auto_grow_enabled is set to true in the module specifically to avoid this — the code notes it was enabled by hand during an earlier remediation, never committed, and then turned back off by a later apply. If you find auto-grow disabled on a running server, the fix is in code, not in the portal.

Replicas restart constantly, or return 500s while staying in rotation

The API's probes are split so that a slow database does not kill replicas: liveness hits /api/v1/health/live (no dependencies, answered on the event loop) and readiness hits /api/v1/health/ready (checks the database, 503 when unreachable). Liveness recycles a process that cannot answer at all; readiness drains a replica whose database is briefly slow instead of killing it.

If you change these paths, keep that split. Pointing liveness at a database-touching endpoint reintroduces the restart storm.

Configuration confusions

allowed_domains and allowed_auth_domains are different variables

• allowed_domains is a comma-separated string, required, and goes to the dashboard container as the ALLOWED_DOMAINS env var.
• allowed_auth_domains is a list of strings, defaults to [], and goes to the auth module.

Setting one does not set the other. An empty ALLOWED_DOMAINS means the application permits every domain, which is why allowed_domains has no default.

azure_ad_client_id is not automatically the app the module creates

azure_ad_client_id and azure_ad_tenant_id are required inputs fed to the containers and the vault. The auth module separately creates an app registration and exposes its client id as module.auth.application_id. They are two different values unless you deliberately make them match — typically by applying once, taking module.auth.application_id, and putting it in tfvars.

Variables that do nothing

Three root variables are declared but never referenced in main.tf:

Variable	Note
location	Every submodule takes its location from the resource group data source
enable_machine_groups	Not wired to anything in this module
enable_business_units	Not wired to anything in this module

deploy_dev is passed to the containers module, but no dev container app is defined there, and frontend_dev_url is hardcoded to null.

container_image mostly does not do what it looks like

The running images are assembled from existing_registry_server (or the ACR login server), frontend_image_name / api_image_name, and the corresponding tag. container_image is passed through to the containers module but only feeds a local value for extracting a name and tag. Changing it does not change what runs.

The database is open to the internet

Passing "0.0.0.0/0" in allowed_ips creates an allow_all firewall rule spanning 0.0.0.0–255.255.255.255. The default is [], which leaves only the allow_azure rule that Azure-hosted services need. Only add the catch-all knowingly, and never in production.

Getting diagnostics

The repo ships scripts/check.ps1 and scripts/status.ps1 for health checks and component status. For anything below that, container logs go to the Log Analytics workspace the Container Apps environment is wired to.

az containerapp logs show --name reportmate-functions-api --resource-group rg-reportmate-prod --follow