Configuring Alerts

How alert notifications are configured in ram290476/investor-dashboard as deployed in production: the per-issue caps on email and SMS, and every other alert setting (recipients, SNS topics, CloudWatch alarms, metric filters, EventBridge rules, structured log events), with steps to add a new alert.

Status (Sun Oct 11 2026, 11:20 AM PT).

PR #153 (email and SMS caps) and PR #157 (the plan-time count fix and the topic layout below) are merged as 78dbb17. Deploy production #200 applied them (10:43–10:46 PT).

Email is capped. SMS is off because alert_sms_e164 is not set.

The existing confirmed email subscriptions were kept, so nobody had to confirm anything again.

PR #158 (one email per CVE across Inspector and Security Hub) is merged as 9a9dba0 and was deployed by Deploy production #201 (11:11–11:15 PT). VULN_WINDOW_MINUTES=1440 is set on the notifier. The scan of #201's image at 11:14 PT sent 16 emails for 16 CVEs and held back the 16 Security Hub copies (alert_suppressed). After #200 the same scan had sent 32.

Contents

Per-issue caps: email and SMS
What happens to repeats
All other alert configuration

1. Per-issue caps: email and SMS

Alert flow

CloudWatch alarms (us-west-1)            ─┐
EventBridge rule aws-health              ─┼─> invdash-ops-alarms ─────────┐
                                          │                               │
EventBridge security rules (8)           ─┼─> invdash-security-alarms ────┼─> invdash-alert-notifier (Lambda, us-west-1)
API key rotation check Lambda            ─┘                               │      │ caps per issue and channel
                                                                          │      │ (DynamoDB invdash-alert-cap)
CloudFront 5xx alarm (us-east-1) ──> invdash-ops-alerts-us-east-1 ────────┘      │
                                                                                 ├─> invdash-ops-alerts ──────> email (alert_emails)
                                                                                 ├─> invdash-security-alerts ─> email (security_alert_emails)
                                                                                 └─> invdash-sms ─────────────> SMS (only when alert_sms_e164 is set)

Intake topics (invdash-ops-alarms, invdash-security-alarms, invdash-ops-alerts-us-east-1): alarms and rules publish here. Their only subscriber is the notifier Lambda.
Delivery topics (invdash-ops-alerts, invdash-security-alerts, and optionally invdash-sms): the inboxes and handset subscribe here. Only the notifier publishes here. These are the original topics, and the email subscriptions created before #153 are still on them.
There are no *-email topics. An earlier version of #153 proposed invdash-ops-email / invdash-security-email. #157 replaced that design before anything was applied.

Each alert issue sends at most 1 email per window and at most 1 SMS per window (default window: one Pacific clock hour). The two channels are counted separately, each with its own window length and maximum, so changing one does not affect the other.

Settings

Set these in infra/terraform/terraform.tfvars, which is git-ignored. In production the deploy workflow writes this file from the TERRAFORM_TFVARS secret. Commented examples are in infra/terraform/terraform.tfvars.example. The variables are declared in infra/terraform/variables.tf, passed through module "alerting" in infra/terraform/main.tf, and declared again in infra/terraform/modules/alerting/sms.tf.

Terraform variable	Default	Allowed	Lambda environment variable	What it limits
`email_window_minutes`	`60`	1–1440	`EMAIL_WINDOW_MINUTES`	Length of the email window (minutes)
`email_max_per_window`	`1`	1–100	`EMAIL_MAX_PER_WINDOW`	Emails per issue per email window
`sms_window_minutes`	`60`	1–1440	`SMS_WINDOW_MINUTES`	Length of the SMS window (minutes)
`sms_max_per_window`	`1`	1–100	`SMS_MAX_PER_WINDOW`	Texts per issue per SMS window
`vuln_window_minutes` (PR #158, deployed)	`1440`	1–1440	`VULN_WINDOW_MINUTES`	Window for vulnerability (`vuln:`) keys on both channels. Replaces the email and SMS windows for CVE/GHSA findings only. 1440 = one Pacific calendar day.
`alert_sms_e164`	`""` (SMS off)	E.164, for example `+14155552671`	`SMS_TOPIC_ARN` is set only when this is set	The handset number. Empty leaves SMS off; email is capped either way.
`alert_notifier_enabled`	`true`	bool	–	Subscribes `invdash-ops-alerts-us-east-1` to the notifier. Must be a plan-time value (#157), because the notifier ARN is unknown until apply. `false` leaves the CloudFront alarm with no email or SMS.
`enable_cloudfront_alarms`	`true`	bool	–	Creates the us-east-1 CloudFront 5xx alarm, its topic and KMS key.

Example terraform.tfvars:

alert_emails         = ["ram.Vellamsetti@gmail.com"]
email_window_minutes = 60 # fixed Pacific clock buckets from local midnight; 60 = one clock hour
email_max_per_window = 1  # emails per issue per window
alert_sms_e164       = "+1XXXXXXXXXX" # optional
sms_window_minutes   = 60 # independent of email_window_minutes
sms_max_per_window   = 1  # texts per issue per SMS window
vuln_window_minutes  = 1440 # one mail per CVE (Inspector + Security Hub merged) per Pacific day

Changes reach production through the Deploy production workflow, which runs on main after CI passes. The Lambda reads the values at cold start. A missing, non-numeric or < 1 value falls back to 60 and 1 (_positive() in sms_notifier.py).

How the window works

Windows are fixed clock buckets in America/Los_Angeles, aligned to local midnight, not a rolling hour from the first message (window_bounds() in sms_cap.py).

With the default 60 minutes, 14:00–14:59 PT is one window and 15:00 starts the next. An alert at 14:50 PT and another at 15:05 PT are in different windows, so both can be sent.
If the window length does not divide 1440, the last bucket of the day is shorter.
The window id is the start time with its UTC offset (for example 2026-10-11T14:00-0700), so the repeated 1 AM hour on a fall-back day counts as two windows.

Where it lives

Piece	Name / path
Notifier Lambda (Terraform)	`infra/terraform/modules/alerting/sms.tf`, `aws_lambda_function.notifier`, function name `invdash-alert-notifier` (`local.sms_fn = "${var.name}-alert-notifier"`), Python 3.12, arm64, handler `sms_notifier.handler`, 15 s timeout, 128 MB
Notifier code	`services/data-jobs/src/functions/sms_notifier/sms_notifier.py` (handler, `DynamoCapStore`, topic routing) and `services/data-jobs/src/functions/sms_notifier/sms_cap.py` (`issue_key()`, `window_bounds()`, `plan()`, `deliver()`, `format_email()`, `format_sms()`)
Tests	`services/data-jobs/tests/test_sms_cap.py`; `infra/terraform/tests/plan_time_count.tftest.hcl` (mock-provider plan, run in CI as `terraform test`, guards against apply-time `count`)
Notifier log group	`/aws/lambda/invdash-alert-notifier` (audit KMS key, `log_retention_days`)
Dedupe store	DynamoDB table `invdash-alert-cap` (`aws_dynamodb_table.cap`): on-demand, hash key `issue_key`, range key `channel` (`email` or `sms`), so there is one item per issue per channel. Attributes `window`, `sent`, `suppressed`, `expires_at`. Data-key encryption, point-in-time recovery, deletion protection.
TTL	Attribute `expires_at` = the end of the following window for that channel (window start + 2 × window length). The item outlives its own window by one window, so the next window's first message can report how many repeats were held back.
Intake topics (notifier inputs)	`invdash-ops-alarms` and `invdash-security-alarms` (`aws_sns_topic.alarm` in `sms.tf`, with `aws_sns_topic_subscription.notifier` and `aws_lambda_permission.topics`). `invdash-ops-alerts-us-east-1` for the CloudFront alarm (`aws_sns_topic_subscription.sms_notifier` in `modules/us_east_1/main.tf`, `count = enable_cloudfront_alarms && alert_notifier_enabled`; permission `aws_lambda_permission.cloudfront_topic`).
Delivery topics (notifier outputs)	`invdash-ops-alerts` and `invdash-security-alerts` (`aws_sns_topic.this` in `modules/alerting/main.tf`, email subscriptions `aws_sns_topic_subscription.email`). `invdash-sms` (`aws_sns_topic.sms`, only when `alert_sms_e164` is set, subscription `aws_sns_topic_subscription.handset`, `protocol = "sms"`, sent as Transactional).
Lambda environment	`CAP_TABLE_NAME`, `OPS_EMAIL_TOPIC_ARN` (= `invdash-ops-alerts`), `SECURITY_EMAIL_TOPIC_ARN` (= `invdash-security-alerts`), `EMAIL_WINDOW_MINUTES`, `EMAIL_MAX_PER_WINDOW`, `SMS_WINDOW_MINUTES`, `SMS_MAX_PER_WINDOW`, `AWS_USE_FIPS_ENDPOINT`, `SMS_TOPIC_ARN` (SMS only), `VULN_WINDOW_MINUTES` (default `1440`)
IAM	Role `invdash-alert-notifier`, inline policy `alert-notifier`: `dynamodb:GetItem` and `dynamodb:UpdateItem` on the cap table only; `sns:Publish` only on the delivery topics (and `invdash-sms` when set); KMS via DynamoDB/SNS/Lambda; logs to its own group
Terraform outputs (root)	`ops_topic_arn` / `security_topic_arn` = intake topics; `ops_email_topic_arn` / `security_email_topic_arn` = delivery topics `invdash-ops-alerts` / `invdash-security-alerts` (these are output names, not topic names); `cloudfront_alarm_topic_arn`; `sms_notifier_arn`

Routing (email_topic_arn() in sms_notifier.py). A message is routed by the name of the intake topic that delivered it. Names ending in -security-alarms are emailed through invdash-security-alerts, to security_alert_emails, or to alert_emails when that list is empty. Everything else goes through invdash-ops-alerts to alert_emails, including invdash-ops-alarms and the us-east-1 CloudFront topic.

Important: subscribe inboxes only to invdash-ops-alerts / invdash-security-alerts, and the handset only to invdash-sms. A subscription on an intake topic (invdash-ops-alarms, invdash-security-alarms, invdash-ops-alerts-us-east-1) bypasses the cap. The topic policies allow only the notifier role, and the account owner, to publish to the delivery topics. AWS Budgets emails come from AWS Budgets directly and are not capped.

How the issue key is derived

issue_key(subject, message) in sms_cap.py returns a stable id for one problem. Alarm state is not part of the key, so ALARM and OK notifications for the same alarm share one budget. Checked in this order:

Source	Key	Example
CloudWatch alarm subject (`ALARM: "<name>"`, `OK: …`, `INSUFFICIENT_DATA: …`)	`alarm:<AlarmName>`	`alarm:invdash-site-api-5xx`
JSON message with `AlarmName`	`alarm:<AlarmName>`
JSON with `job` and `code` / `error_code` / `error` (top level or in `detail`)	`job:<job>:<code>`
GuardDuty finding	`guardduty:<detail.type>`
Vulnerability finding (CVE or GHSA), from Inspector or Security Hub (PR #158, deployed)	`vuln:<ID>:<resource>`. The ID is Inspector's `packageVulnerabilityDetails.vulnerabilityId` or Security Hub's `Vulnerabilities[0].Id`; if those are missing, the CVE/GHSA id in the title. The resource is `ecr:<repositoryName>` or `lambda:<functionName>`; any other resource uses its id without the image digest. The source product is not part of the key, so the Inspector finding and its Security Hub copy share one counter. Uses `vuln_window_minutes` (default one Pacific day).	`vuln:CVE-2026-56857:ecr:invdash/jobs`
Security Hub finding	`securityhub:<first finding Title>`, only for findings without a vulnerability id (for example a control such as IAM.6). CVE/GHSA findings use the `vuln:` key above.	`securityhub:IAM.6 Hardware MFA should be enabled for the root user`
Inspector finding	`inspector:<detail.title or findingArn>`, only for findings without a vulnerability id (for example network reachability).
CloudTrail API event (IAM, S3, audit tampering, API key change)	`event:<eventName>:<requestParameters.name or userName>`	`event:PutParameter:/invdash/api-keys/…`
AWS Health event	`health:<eventTypeCode>`
Plain-text alarm body (`- Name: <alarm>`)	`alarm:<name>`
API key rotation reminder (subject contains "API key rotation")	`key-rotation`
Anything else	`msg:<first 16 hex of SHA-256(subject + first 180 chars of body)>`

Keys are whitespace-normalized and clipped to 240 characters.

Why vulnerability findings are merged (PR #158, deployed in #201). When Deploy production #200 pushed a new jobs image at 10:45 PT on Oct 11 2026, the scan reported 16 CVEs. Each one came twice: as an Inspector2 Finding event and as its Security Hub Findings - Imported copy. Under the old keys (inspector:<title> and securityhub:<title>) that was 32 security emails in two minutes. With PR #158 each CVE in a repository is one issue:

16 emails for that scan, one per CVE.
Within the same Pacific day, rescans and new image pushes with the same CVE send nothing more. Every new image gets a new Inspector finding, which is why the digest is not part of the key.
The next day's first email for that CVE shows how many repeats were held back.
Because the window starts at midnight, a CVE seen at 23:50 PT and again at 00:10 PT sends 2 emails.
A different CVE, or the same CVE in a different repository or function, is a separate issue.

Tests: services/data-jobs/tests/test_sms_cap.py with fixtures tests/fixtures/notifier/inspector2_finding_cve.json and securityhub_imported_cve.json. These are the 10:46 PT events for CVE-2026-56857 with the account id redacted.

2. What happens to repeats

Within one window, after an issue has used its *_max_per_window messages on a channel:

Repeats are suppressed on that channel: no email and/or no text. Email and SMS are decided independently, and email is attempted before SMS. With the defaults, a flapping alarm sends one email and one text per clock hour instead of one per ALARM/OK transition.
They are still logged. Each suppressed repeat increments suppressed on the DynamoDB item and writes a JSON line to /aws/lambda/invdash-alert-notifier. The original alarm/event is also unchanged: CloudWatch alarm history, EventBridge, and the source Lambda logs (for example the prefs_api_error lines in section 3.6) still record every occurrence.
The next allowed message carries the suppressed count. The first message for that issue in a later window includes the count from the previous window:
- Email: the subject gets (suppressed N repeats) (or 1 repeat), kept within SNS's 100-character subject limit, and the body ends with Suppressed N repeats in the previous window.
- SMS: the text (up to 140 characters of subject/summary) gets (suppressed N repeats), capped at 300 characters. The count only carries over from the window immediately before (the item expires after that), and only if the issue recurs.
Fail open. If the cap table cannot be read or updated, that channel is sent anyway and alert_cap_failed_open is logged. A failed SNS publish does not use up the slot; the invocation raises so SNS retries it.

Notifier log events (one JSON object per line, from _log() in sms_cap.py):

`event`	Fields	Meaning
`alert_sent`	`issue_key`, `channel`, `window`, `suppressed_prior`, `reason`	Message published and counted
`alert_suppressed`	`issue_key`, `channel`, `suppressed` (count so far in this window), `window`	Repeat held back
`alert_cap_failed_open`	`issue_key`, `channel`, `error` (`store_read_failed` or `store_suppress_failed`)	Store error; sent anyway
`alert_cap_record_failed`	`issue_key`, `channel`, `window`	Sent, but the counter could not be updated

{"event": "alert_suppressed", "issue_key": "alarm:invdash-site-api-5xx", "channel": "email", "suppressed": 3, "window": "2026-10-11T14:00-0700"}

Logs Insights query for suppressed repeats (log group /aws/lambda/invdash-alert-notifier):

fields @timestamp, issue_key, channel, suppressed, window
| filter event = "alert_suppressed"
| sort @timestamp desc

3. All other alert configuration

Everything below is read from main at 9a9dba0 (Oct 11 2026) and matches production after Deploy production #201. Resource names use the project prefix invdash (Terraform variable project, default "invdash") and the primary region us-west-1.

3.1 Email recipients

Setting	File	What it does
`alert_emails` (list, required)	`infra/terraform/variables.tf`, set in `terraform.tfvars`	Email subscribers on the ops delivery topic `invdash-ops-alerts`. Ops alarms and the CloudFront alarm reach them through the notifier. Also the monthly budget recipients (`budget_emails = var.alert_emails` in `infra/terraform/main.tf`).
`security_alert_emails` (list, default `[]`)	`infra/terraform/variables.tf`	Email subscribers on the security delivery topic `invdash-security-alerts`. When empty, `local.security_alert_emails` in `main.tf` falls back to `alert_emails`.

The current recipient is ram.Vellamsetti@gmail.com. Its subscriptions on invdash-ops-alerts and invdash-security-alerts are Confirmed, checked after Deploy production #200 at 10:47 PT. Each address confirms two subscriptions, one per delivery topic. Nothing is subscribed by email in us-east-1: #153 removed the old direct email subscription on invdash-ops-alerts-us-east-1, and CloudFront mail now arrives from invdash-ops-alerts. An unconfirmed subscription receives nothing.

To change recipients: edit alert_emails / security_alert_emails in the tfvars (the TERRAFORM_TFVARS secret for production) and deploy. Each new address then confirms its subscription emails.

3.2 SNS topics

Topic	Role	Defined in	Region	Publishers	Subscribers
`invdash-ops-alarms`	intake	`modules/alerting/sms.tf` (`aws_sns_topic.alarm["ops"]`)	us-west-1	All CloudWatch alarms in `modules/observability` and `modules/user_prefs` (alarm and OK actions); EventBridge rule `aws-health`	`invdash-alert-notifier` only
`invdash-security-alarms`	intake	`modules/alerting/sms.tf` (`aws_sns_topic.alarm["security"]`)	us-west-1	The 8 security EventBridge rules (below); the API key rotation check Lambda (`modules/api_keys`, env `TOPIC_ARN`)	`invdash-alert-notifier` only
`invdash-ops-alerts-us-east-1`	intake	`modules/us_east_1/main.tf` (`aws_sns_topic.ops`)	us-east-1	CloudFront 5xx alarm only. Created when `enable_cloudfront_alarms = true` (default); own KMS key `alias/invdash-alerts`	`invdash-alert-notifier` only (cross-region Lambda subscription)
`invdash-ops-alerts`	delivery	`modules/alerting/main.tf` (`aws_sns_topic.this["ops"]`)	us-west-1	`invdash-alert-notifier` only	`alert_emails` (email)
`invdash-security-alerts`	delivery	`modules/alerting/main.tf` (`aws_sns_topic.this["security"]`)	us-west-1	`invdash-alert-notifier` only	`security_alert_emails` (email)
`invdash-sms`	delivery	`modules/alerting/sms.tf` (`aws_sns_topic.sms`)	us-west-1	`invdash-alert-notifier` only	`alert_sms_e164`. Not created today (SMS off).

All topics are KMS-encrypted (the data key, or the us-east-1 alerts key) and deny publishes over plain HTTP. Intake topics allow cloudwatch.amazonaws.com and events.amazonaws.com to publish only from this account; invdash-ops-alerts-us-east-1 allows CloudWatch only. Delivery topics allow publishes only from the notifier role and the account owner.

3.3 CloudWatch alarms

Alarm names are invdash-<key>. Every us-west-1 alarm sends both ALARM and OK notifications (alarm_actions and ok_actions) to invdash-ops-alarms. ALARM and OK for the same alarm share one issue key, so a flapping alarm sends at most one email per window instead of one per transition.

infra/terraform/modules/observability/main.tf (local.base_alarms, local.canary_alarm_defs, resource aws_cloudwatch_metric_alarm.this). Custom metrics are in namespace InvestorDashboard (metrics_namespace), dimension service = invdash, emitted by services/data-jobs/src/app/observability.py and dashboard_build.

Alarm	Metric	Fires when	What it means
`invdash-lambda-errors`	`AWS/Lambda Errors` (account-wide, Sum)	≥ 1 in 2 consecutive 15-minute periods	A job Lambda is erroring repeatedly. Check the failing function's log group.
`invdash-lambda-throttles`	`AWS/Lambda Throttles` (Sum)	≥ 1 in one 15-minute period	Concurrency limit hit; runs are being throttled.
`invdash-dlq-not-empty`	`AWS/SQS ApproximateNumberOfMessagesVisible` on the jobs DLQ (Max)	> 0 in 5 minutes	A job run failed all retries and is waiting in the dead-letter queue.
`invdash-failed-runs`	`FailedRuns` (Sum)	≥ 2 in one hour	Two or more source fetches failed (counted by `source_run()`). Use the saved query `invdash/failed-source-runs`.
`invdash-stale-p1-source`	`StaleP1Sources` (Max)	> 0 for 2 hours in a row	A P1 data source is older than its tier's max age.
`invdash-rate-limit-headroom`	`RateLimitHeadroomPct` (Min)	< 20% in one hour	A provider's free-tier quota is more than 80% used.
`invdash-freshness-slo-burn`	`FreshP1Ratio` (Avg, 6 h)	< 0.94 (1 − 6 × 1%)	Freshness SLO (99%) burning 6x too fast.
`invdash-dashboard-payload-size`	`DashboardRawBytes` (Max, 6 h)	> 4,194,304 bytes	`serving/dashboard.json` is over 4 MiB, approaching the 6 MiB Lambda response limit.
`invdash-canary-failing` *	`CloudWatchSynthetics SuccessPercent`	< 100 for 2 canary runs	External health check failed twice (site down or data stale).
`invdash-availability-fast-burn` *	same	< 92.8% over 1 hour	Availability SLO (99.5%) burning 14.4x too fast.
`invdash-availability-slow-burn` *	same	< 97% over 6 hours	Availability SLO burning 6x too fast.

* Canary alarms exist only when health_url is set (infra/terraform/variables.tf, default ""). It is empty in production, so the canary and these three alarms are off (15 alarms exist in us-west-1 today).

infra/terraform/modules/user_prefs/main.tf (site API / preferences Lambda):

Alarm	Metric	Fires when	What it means
`invdash-site-api-5xx`	`AWS/ApiGateway 5xx` (ApiId, Stage `$default`)	≥ 1 in 2 consecutive 5-minute periods	The site API returned server errors.
`invdash-prefs-api-sts-unavailable`	`invdash/prefs-api STS_UNAVAILABLE`	≥ 1 in 5 minutes	The API could not assume the preferences role (STS).
`invdash-prefs-api-serving-read-failed`	`invdash/prefs-api SERVING_READ_FAILED`	≥ 1 in 5 minutes	The API could not read a serving object (S3 or KMS).
`invdash-prefs-api-prefs-store-unavailable`	`invdash/prefs-api PREFS_STORE_UNAVAILABLE`	≥ 1 in 5 minutes	The API could not reach the preferences DynamoDB table.
`invdash-prefs-api-serving-document-invalid`	`invdash/prefs-api SERVING_DOCUMENT_INVALID`	≥ 1 in 5 minutes	A serving document was not valid JSON.
`invdash-prefs-api-internal`	`invdash/prefs-api INTERNAL`	≥ 1 in 5 minutes	Unexpected error in the API.
`invdash-prefs-api-ticker-added-publish-failed`	`invdash/prefs-api TICKER_ADDED_PUBLISH_FAILED`	≥ 1 in 5 minutes	Preferences saved, but the `TickerAdded` event (backfill) could not be published.

infra/terraform/modules/us_east_1/main.tf:

Alarm	Metric	Fires when	What it means
`invdash-cloudfront-5xx`	`AWS/CloudFront 5xxErrorRate` (Region `Global`)	> 5% for 3 × 5 minutes	Viewers are getting server errors from CloudFront. Publishes to `invdash-ops-alerts-us-east-1`, which invokes the notifier; mail arrives from `invdash-ops-alerts`. Look for the alarm in the us-east-1 console.

3.4 Metric filters (logs → metrics)

The only log metric filters are in infra/terraform/modules/user_prefs/main.tf: local.prefs_api_error_alarms drives both aws_cloudwatch_log_metric_filter.prefs_api_error and aws_cloudwatch_metric_alarm.prefs_api_error (one filter and one alarm per code).

Log group: the site API function's group (aws_cloudwatch_log_group.api_fn).
Filter name: invdash-prefs-api-<code in lower-kebab>, for example invdash-prefs-api-serving-read-failed.
Pattern: a JSON filter, { ($.event = "<event>") && ($.code = "<CODE>") }. A quoted-term pattern such as "event": "x" is rejected by PutMetricFilter, and services/data-jobs/tests/test_prefs_api_metric_filters.py checks the filters against what _log_event prints.
Metric: name <CODE>, namespace invdash/prefs-api, value 1 per matching line.

Code (map key)	`event` matched
`STS_UNAVAILABLE`, `SERVING_READ_FAILED`, `PREFS_STORE_UNAVAILABLE`, `SERVING_DOCUMENT_INVALID`, `INTERNAL`	`prefs_api_error`
`TICKER_ADDED_PUBLISH_FAILED`	`ticker_added_publish_failed`

modules/observability/main.tf has no metric filters. Job metrics come from Powertools embedded metrics (EMF) in services/data-jobs/src/app/observability.py, not from log filters. It also defines three saved Logs Insights queries: invdash/failed-source-runs, invdash/slowest-sources, invdash/trace-one-run.

3.5 EventBridge rules (`infra/terraform/modules/alerting/main.tf`)

AWS service events on the default bus are free, so these replace metric-filter alarms for security events. Rule name: invdash-<key>. Each rule's target is the intake topic shown.

Rule	Target topic	Fires on
`invdash-iam-changes`	`invdash-security-alarms`	IAM user, access key, login profile, policy, trust-policy and MFA changes
`invdash-risky-sign-in`	`invdash-security-alarms`	Root console sign-in, or any console sign-in without MFA
`invdash-guardduty-findings`	`invdash-security-alarms`	GuardDuty findings, severity ≥ 4 (medium and above)
`invdash-securityhub-high`	`invdash-security-alarms`	New, active Security Hub findings rated HIGH or CRITICAL
`invdash-inspector-high`	`invdash-security-alarms`	Active Inspector findings rated HIGH or CRITICAL
`invdash-audit-tampering`	`invdash-security-alarms`	Attempts to stop or weaken CloudTrail, Config, GuardDuty, Security Hub, KMS or Inspector
`invdash-s3-exposure`	`invdash-security-alarms`	Bucket policy, ACL, public access block, Object Lock or replication changes
`invdash-api-key-changed`	`invdash-security-alarms`	SSM `PutParameter`/`DeleteParameter`/`LabelParameterVersion` under `/invdash/api-keys/`
`invdash-aws-health`	`invdash-ops-alarms`	Any AWS Health event

IAM and root/global sign-in events only reach the us-east-1 bus. modules/us_east_1/main.tf forwards them to the us-west-1 default bus (output global_event_patterns), where iam-changes and risky-sign-in alert.

securityhub-high and inspector-high both fire for the same CVE, because Security Hub imports Inspector's findings. Since PR #158 (deployed in #201), both copies share one vuln:<ID>:<resource> key with a one-day window. Each CVE now emails once per Pacific day per repository: 16 emails for #201's image scan at 11:14 PT, where #200's scan at 10:46 PT had sent 32.

Also on the security intake topic: the API key rotation check Lambda (modules/api_keys/main.tf, code services/data-jobs/src/functions/key_rotation_check/key_rotation_check.py) runs daily (check_schedule). It publishes to invdash-security-alarms (issue key key-rotation) when a key is overdue, due soon (remind_days), or never stored (Mondays). The notifier then emails it through invdash-security-alerts.

Budget alerts (modules/observability/main.tf, aws_budgets_budget.monthly): AWS Budgets emails alert_emails directly at 80% actual and 100% forecast of monthly_budget_usd (default 40). These do not go through SNS and are not capped.

3.6 Log events and structured `_log_event` JSON

Site API (services/data-jobs/src/functions/prefs_api/prefs_api.py): _log_event(event_name, route, method, exc, code) prints one JSON object per line with fixed keys only (no request body, token, or exception text):

{"event": "prefs_api_error", "route": "GET /dashboard", "method": "GET", "error_class": "ClientError", "code": "SERVING_READ_FAILED"}

event: prefs_api_error (handler failures) or ticker_added_publish_failed (from _queue_backfill).
route: normalized, for example GET /prefs, GET /chart/{ticker}, GET /stock/{ticker}, or GET /unknown.
method: an HTTP verb or UNKNOWN.
error_class: class of the underlying cause (exc.__cause__) or of the exception itself.
code: STS_UNAVAILABLE, SERVING_READ_FAILED, PREFS_STORE_UNAVAILABLE (from _client_failure by AWS operation), SERVING_DOCUMENT_INVALID, INTERNAL, or TICKER_ADDED_PUBLISH_FAILED.

Jobs (services/data-jobs/src/app/observability.py, Powertools logger): source_run (per source: source_id, started_at, outcome, rows, duration_ms, error_type, error), rate_headroom (provider, used, limit) and freshness (stale, stale_p1, p1_due). The saved queries filter on these.

Key rotation check: key_rotation_status (one per key) and key_rotation_summary (emailed plus counts). Logs are kept 400 days as rotation evidence.

Log retention is log_retention_days (default 400) and log groups are encrypted with the audit KMS key.

3.7 How to add a new alert

A. New error code in the site API (log line → filter → alarm):

In prefs_api.py, call _log_event("<event>", route, method, exc, "<NEW_CODE>") where the failure happens. Keep the fixed keys and add no free text.
In infra/terraform/modules/user_prefs/main.tf, add an entry to local.prefs_api_error_alarms:
```
NEW_CODE = {
  event       = "prefs_api_error"
  description = "Site API <what went wrong>"
}
```
This creates the filter invdash-prefs-api-new-code, the metric invdash/prefs-api NEW_CODE, and an alarm on the ops intake topic (invdash-ops-alarms).
Extend services/data-jobs/tests/test_prefs_api_metric_filters.py if needed, then run pytest.

B. New metric alarm on a job or AWS metric:

Emit the metric (for jobs, metrics.add_metric(name="X", unit=..., value=...) through app/observability.py so it gets the service dimension), or pick an existing AWS metric.
Add an entry to local.base_alarms in infra/terraform/modules/observability/main.tf with description, namespace, metric, stat, dims (a jsonencode(...)), period, evals, datapoints, op, threshold and missing. aws_cloudwatch_metric_alarm.this creates invdash-<key>, wires it to the ops intake topic (invdash-ops-alarms) and adds it to the ops dashboard's alarm widget.

C. New AWS event alert: add an entry to local.rules in infra/terraform/modules/alerting/main.tf with topic (ops or security), description and an EventBridge pattern. For an IAM or global sign-in event, put it in local.global_rules so us-east-1 forwards it.

Then: terraform fmt, terraform validate, terraform plan in infra/terraform, and deploy through the Deploy production workflow (its destroy gate checks the plan). New alarms and rules publish to the intake topics (invdash-ops-alarms / invdash-security-alarms), so the new alert goes through invdash-alert-notifier and is capped per issue and channel automatically (section 1). The issue key is the alarm name, or the event fields in How the issue key is derived. Never subscribe an inbox or phone to an intake topic or to invdash-ops-alerts-us-east-1.

Navigation Menu

Repository navigation

Configuring Alerts

Configuring Alerts

1. Per-issue caps: email and SMS

Alert flow

Settings

How the window works

Where it lives

How the issue key is derived

2. What happens to repeats

3. All other alert configuration

3.1 Email recipients

3.2 SNS topics

3.3 CloudWatch alarms

3.4 Metric filters (logs → metrics)

3.5 EventBridge rules (`infra/terraform/modules/alerting/main.tf`)

3.6 Log events and structured `_log_event` JSON

3.7 How to add a new alert

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Clone this wiki locally

Configuring Alerts

Configuring Alerts

1. Per-issue caps: email and SMS

Alert flow

Settings

How the window works

Where it lives

How the issue key is derived

2. What happens to repeats

3. All other alert configuration

3.1 Email recipients

3.2 SNS topics

3.3 CloudWatch alarms

3.4 Metric filters (logs → metrics)

3.5 EventBridge rules (infra/terraform/modules/alerting/main.tf)

3.6 Log events and structured _log_event JSON

3.7 How to add a new alert

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Clone this wiki locally

3.5 EventBridge rules (`infra/terraform/modules/alerting/main.tf`)

3.6 Log events and structured `_log_event` JSON