Scraped to death: the 2026-09-26 forums outage

[DanVanAtta-bot]

On Saturday 2026-09-26 the forum was slow or down from about 14:50 to 16:14 UTC. A crawler sent about 70 requests a second from ~27,000 rotating IP addresses, scraping the whole forum. Call it a free load test: it found a database index we never had. What brought the site back is that the crawler stopped, or at least slowed down a lot. By then we had a set of fixes ready, and they're all deployed now.

What happened

We migrated the forum from MongoDB to Postgres. NodeBB converted the data itself, but it only creates its database indexes on a fresh install, so our migrated database never got them. The one that mattered here is on expireAt in the legacy_object table. That table has about 450,000 rows, one for every "thing" on the forum.

NodeBB's Postgres support is really a Redis-style key store built on top of Postgres. Before every write, it deletes expired rows from legacy_object, which means reading expireAt, the column with no index. So every write came with a full scan of 450k rows.

We also had guest visitor counting turned on. To count unique guests, NodeBB records a visitor's IP address the first time it sees it each hour, and that's a write.

Put those together:

• Traffic: about 70 requests a second, roughly 10x normal. Every page request meant database reads and a session update.
• Full scans: only the requests that wrote something paid for a scan, mostly each rotating IP's first visit of the hour. We measured about 170 full scans of legacy_object a minute, roughly 3 a second.

Three writes a second doesn't sound like much, but each one read all 450k rows, on a server with one CPU. That pegged it at 100%. The rest of the traffic queued up behind the scans, nginx ran out of connection slots, and visitors got hard errors.

To make it worse, every request for assets (JS, CSS, fonts) also hit the database. Each one was treated as an existing guest session, and NodeBB updates a session's last-activity time on every request.

Timeline (UTC)

Time
14:50	The crawler starts paging deep into the biggest threads. Pages start stalling.
14:54	Worst minute: nginx runs out of connections and returns errors.
before 16:14	We watch it for about half an hour and put fixes together. The server is at 100% CPU, too busy to take a deploy.
16:14	We turn nginx off to cut the traffic so the deploy can get through. The crawler slows to a trickle the same minute.
16:23	nginx is back with the first fixes. The site is up.
evening	The rest of the fixes go out. The crawler comes back twice (17:00, 20:02), much smaller.

Why the crawler slowed the moment nginx went off, we don't know. Its earlier bursts also stopped on their own after 1.5–2.4 hours.

What we changed

• An index on expireAt, so that delete is no longer expensive. Full scans of the table went from about 170 a minute to 4 in 9 minutes. The forum now creates its indexes on every start, so a migrated or restored database can't run without them again.
• Guest view counting is off, so a guest page view doesn't issue the write and the delete in the first place, even though the delete is cheap now.
• nginx serves assets from its own cache instead of through NodeBB, so an asset request no longer touches the database.
• Most categories are hidden from guests. Only news and bug reports are public; everything else needs a free account. This is permanent.
• Load protection: a cap on concurrent page renders (overload gets a fast "busy" page instead of a hang), per-IP rate limits, and nginx's connection pool raised from 768 to 4096.
• Tuning and upkeep: Postgres tuned for our one CPU, the session table made UNLOGGED (halving its share of database time), NodeBB upgraded to 4.16.0, and deploys now wait for a health check.

[DanVanAtta]

Couple bug reports sent to NodeBB:

• missing index: Mongo→Postgres conversion does not create the indexes on legacy_object.expireAt and legacy_zset NodeBB/NodeBB#14926
• smaller - every 404 is listed in DB and has no automated cleanup: errors:404 sorted set grows without bound; only a manual ACP clear prunes it NodeBB/NodeBB#14927