Incident Report: Apr. 26, 2021

[dyc3]

At about 14:10 UTC, OTT started crashing repeatedly.

At about 20:00 UTC, I noticed.

Logs:

Apr 26 15:19:17 opentogethertube app/web.1 /app/roommanager.js:1143
Apr 26 15:19:17 opentogethertube app/web.1 				throw err;
Apr 26 15:19:17 opentogethertube app/web.1 				^
Apr 26 15:19:17 opentogethertube app/web.1 ReplyError: OOM command not allowed when used memory > 'maxmemory'.
Apr 26 15:19:17 opentogethertube app/web.1     at parseError (/app/node_modules/redis-parser/lib/parser.js:179:12)
Apr 26 15:19:17 opentogethertube app/web.1     at parseType (/app/node_modules/redis-parser/lib/parser.js:302:14) {
Apr 26 15:19:17 opentogethertube app/web.1   command: 'SET',
Apr 26 15:19:17 opentogethertube app/web.1   args: [
Apr 26 15:19:17 opentogethertube app/web.1     'rooms',

Theory

An abnormal amount of rooms were active, and attempting to save to redis failed due to the limited memory available in heroku's redis free tier.

• Error in log supports this
• Logs also list out what it was trying to save to redis
  • Most of the rooms available are isTemporary=false, and visibility=unlisted
  • Indicate a deliberate attempt to overload OTT, since the default visibility is public
• cloudflare shows an extreme spike in traffic at around 14:00 UTC

Confirmed

Fix

• enable under attack mode in cloudflare
• cleared the rooms key from redis

Service restored: 20:14 UTC

Future mitigation

• Don't save rooms to redis if there are no users in the room
• More restrictive rate limiting for /api/room/create endpoint (related: Replace current rate limiting with something a bit more flexible. #243)
• Don't crash if saving to redis fails
• Don't save entire JSON blobs of rooms, just all that is necessary to maintain room state. The rest can be loaded from postgres (for permanent rooms).

Cleanup

• delete all unlisted permanent rooms created after 14:00 UTC.

[dyc3]

[dyc3]

Post Mortem

So I've come to realize what the actual problem was. The problem was that my redis did not have key eviction set up. As a result, when it filled up, saving new keys would fail. To mitigate this in the future, I've set up key eviction to evict keys based on the time to live.