AWS Billing Alerting

How Hack for LA finds out that its AWS spend has moved. This page is the record of that configuration, because none of it exists in Terraform — the organization account is configured by hand, so if it is not written down here it is not written down anywhere.

If you only need one sentence: all billing alerting lives in the organization account 504695638931, as AWS Budgets, and notifies hackforla@hackforla.org.

Where billing alerting lives, and why

All billing alerts and budgets belong in the organization (payer) account 504695638931. Never create them in a member account.

The reason is not just tidiness. Hack for LA uses AWS consolidated billing, and under consolidated billing the AWS/Billing EstimatedCharges metric is published only in the payer account — where it already covers every member account. A CloudWatch billing alarm created in a member account has no metric to read. It does not fail loudly; it sits at INSUFFICIENT_DATA forever, looking configured while alerting on nothing.

AWS Budgets are the exception, and that is exactly why the rule needs stating. Unlike billing alarms, Budgets do work inside a member account. So it is entirely possible to create one there, have it appear to work, and end up with billing alerting nobody else can see. That is not hypothetical — the incubator account had precisely that, notifying one individual's personal email address, and it was removed as part of consolidating this configuration.

Per-account visibility comes from a LinkedAccount cost filter on a budget in the organization account, never from a budget in the member account itself.

The account layout

Account	Name	Role
504695638931	hackforla	Management / payer — all billing alerting lives here
035866691871	incubator	Member — the shared infrastructure most projects deploy onto
470363915259	foodoasis	Member

Organization o-x3897m6l40, FeatureSet ALL (consolidated billing).

Billing metrics and alerting are always in us-east-1, regardless of what region an account normally works in. The incubator infrastructure runs in us-west-2, but its billing data is not there. Looking in the wrong region is the most common reason someone concludes that no alerting exists.

What is currently configured

Three AWS Budgets, all in account 504695638931, all monthly, all publishing to the same SNS topic.

Budget	Limit / month	Scope
HfLA Consolidated Monthly Cost	$520	Everything — no cost filter
Incubator Monthly Cost	$350	LinkedAccount = 035866691871
FoodOasis Monthly Cost	$50	LinkedAccount = 470363915259

Each has the same three notifications:

• Actual > 85% — early warning that the month is running hot
• Actual > 100% — the budget has been exceeded
• Forecast > 100% — AWS projects the month will end over budget

There are no CloudWatch billing alarms, and no Cost Anomaly Detection monitors.

Pending, as of 2026-08-10: one CloudWatch alarm, Monthly Billing Alarm (EstimatedCharges >= $210), has not yet been deleted and is still active. It trips every month by design and then stays in ALARM for weeks. If you receive an alert naming it, that is this known leftover, not a new problem. Its removal is tracked on devops#175.

Why these numbers

The thresholds are deliberate, and this is the record of what they were based on — so that the next person can tell a considered number from a stale one.

They were set against actual spend in mid-2026, after the incubator's RDS PostgreSQL 13 Extended Support charge (roughly $149/month) ended. That charge stopped when the shared database was upgraded to PostgreSQL 15; before the upgrade it accounted for over a third of the incubator's bill, so any threshold set earlier will look strangely high.

Scope	Typical monthly spend	Budget	Of budget
Incubator	~$285	$350	~81%
FoodOasis	~$40	$50	~79%
Consolidated	~$425	$520	~82%

The organization account's own spend is about $100/month, almost entirely the AWS Business Support plan.

Two consequences worth understanding before you change anything:

• The 85% notifications are intentionally sensitive. Everything sits at roughly 80% of its limit, so a ~5% rise in spend triggers one. That is the point — it is an early warning, not an emergency. The incubator budget is expected to cross 85% in months with a domain renewal, perhaps once or twice a year.
• FoodOasis has the least headroom — only about $3 between typical spend and its 85% notification. If that account starts doing anything new, this is the first budget that will need revisiting.

The configuration this replaced is worth knowing about, because it is the failure mode to avoid repeating. There were two CloudWatch alarms on EstimatedCharges. One applied an anomaly-detection band to it, which fired ten times in a month — EstimatedCharges is a cumulative counter that resets on the 1st, and anomaly detection is not built for that shape. The other was a static >= $210 threshold that real spend passed early every month, so it spent most of its life in ALARM. Neither could tell anyone anything, and both trained people to ignore billing email. Budgets handle the monthly reset natively, which is the core reason they replaced the alarms.

Who receives alerts, and how to change that

All notifications publish to one SNS topic:

arn:aws:sns:us-east-1:504695638931:Monthly_Org_Billing_Alarm

It currently has a single subscriber: hackforla@hackforla.org (protocol email).

The chain is: a budget crosses a threshold → AWS Budgets publishes to the topic → SNS delivers to every confirmed subscriber. Routing everything through one topic means recipients are changed in one place and apply to all three budgets at once. Do not add recipients to individual budgets.

To see who currently receives alerts:

aws sns list-subscriptions-by-topic \
  --topic-arn arn:aws:sns:us-east-1:504695638931:Monthly_Org_Billing_Alarm \
  --region us-east-1

To add a recipient:

aws sns subscribe \
  --topic-arn arn:aws:sns:us-east-1:504695638931:Monthly_Org_Billing_Alarm \
  --protocol email --notification-endpoint someone@hackforla.org \
  --region us-east-1

A new email subscription does not work until it is confirmed. AWS sends a confirmation link to the address; until someone clicks it, the subscription exists but silently delivers nothing. You can tell the difference in the output of list-subscriptions-by-topic: a confirmed subscription has a real ARN, while an unconfirmed one shows PendingConfirmation in that field. Always check after adding someone.

To remove a recipient, take the SubscriptionArn from the listing and:

aws sns unsubscribe --subscription-arn <arn> --region us-east-1

Prefer an organization address or a distribution list over a personal one. A personal address means alerting quietly depends on one person still being around and still reading that inbox.

One non-obvious permission

For a budget to publish to an SNS topic, the topic's access policy must allow the budgets.amazonaws.com service principal to sns:Publish. The default policy AWS creates with a topic does not — it permits principals within the account, which covers CloudWatch alarms but not Budgets.

The Monthly_Org_Billing_Alarm topic already has this, as a statement named AWSBudgetsSNSPublishingPermissions. You only need to know about it if you create a new topic, in which case budget creation will fail validation until the statement is added. It fails at creation time rather than silently at notification time, which is the one merciful thing about it.

Adding a new member account

When a new AWS account joins the organization, create its budget in the organization account with a LinkedAccount cost filter. Create nothing in the new account itself.

1. Look at what the account actually spends first — aws ce get-cost-and-usage grouped by LINKED_ACCOUNT from the payer account — rather than guessing a limit. A budget set below normal spend fires immediately and permanently, which is how billing alerts become noise.
2. Create the budget with a LinkedAccount filter on the new account ID, with the same three notifications (actual > 85%, actual > 100%, forecast > 100%) pointing at Monthly_Org_Billing_Alarm.
3. Raise the consolidated budget by roughly the new account's expected spend, or it will start reporting over-budget for a reason that has nothing to do with anything going wrong.
4. Confirm the budget appears in aws budgets describe-budgets --account-id 504695638931 and that its notifications list the SNS topic.

Also check the new account for billing configuration it brought with it — its own budgets, and CloudWatch alarms in us-east-1. Anything found there should be removed once the organization-account budget is live, so there is never a window with no coverage.

What to do when an alert fires

An alert means spend crossed a line. It does not say what caused it. Work outward from the account to the service to the resource.

1. Establish whether it is real. An actual > 85% notification early in the month is more interesting than the same alert on the 28th, when it may just be a normal month finishing normally. A forecast alert is a projection, and it is unreliable in a month where something large started or stopped partway through — it extrapolates from what has happened so far.

2. Find which account moved. From the payer account:

aws ce get-cost-and-usage \
  --time-period Start=2026-06-01,End=2026-09-01 \
  --granularity MONTHLY --metrics UnblendedCost \
  --group-by Type=DIMENSION,Key=LINKED_ACCOUNT

3. Find which service moved, by adding a filter for that account and grouping by SERVICE instead. Compare against previous months — a single month in isolation will not tell you what is unusual.

4. Narrow to the specific charge by grouping by USAGE_TYPE and filtering to the service. This is the step that names the actual thing, and the names are specific enough to act on — an unattached load balancer, an oversized instance, or a support charge nobody knew about will each show up here by name.

All of this is also available in the Cost Explorer console in the payer account, which is easier for browsing and worse for sharing what you found.

5. Tell someone. Post in #ops on Slack and raise an issue on hackforla/devops with what you found. If the spend belongs to a specific project's resources, tell that project's team too — the incubator account is shared, so a cost spike there usually belongs to one project rather than to everyone.

If the alert was wrong rather than the spend, fix the threshold and record why here. A budget that fires every month teaches people to ignore billing email, which is worse than having no alerting at all — that is the exact failure this configuration was built to correct.

Related

• devops#175 — consolidating billing alerting into the organization account; the change this page documents
• incubator#118 — the research behind it, including what the previous configuration did and why it was replaced
• AWS: consolidated billing
• AWS: managing costs with Budgets
• AWS: cost filters for linked accounts