Skip to content
Browse files

postmortem: Login Token Mismatch

  • Loading branch information...
znarf committed Mar 26, 2019
1 parent 4761e82 commit b3f0a99f8715330efee10ac30c01ebcd77840c60
Showing with 59 additions and 0 deletions.
  1. +59 −0 postmortem/
@@ -0,0 +1,59 @@
Login Token Mismatch


Background: To approve expenses or approve collectives, we're sending notification emails. As a commodity, these emails were containing in approve/reject links a token to authenticate users in case they were not already authenticated.

When sending these emails, the token was the same for all admins, leading to admins being potentially authenticated as one of the other fellow admins.


All hosts and all collectives with multiple admins were impacted.


(Berlin, UTC+1)

- **2019-03-25 10:15**:
we're contacted on our #support channel by a user reporting the problem, the investigation starts immediately.
- **2019-03-25 11:30**:
the root cause is identified, a misuse of scope and variables in the notification code is discovered.
- **2019-03-25 12:00**:
PR is issued, merged and deployed.
- **2019-03-26 09:00**:
follow up PR is issued, merged and deployed.
- **2019-03-26 09:30**:
JWT secret is rotated, invalidating all authentication tokens generated before this time.

Root Cause

Misuse of variables in the notification code after a refactoring.


- **API**: fixing the code and deploying
- **API**: rotating JWT secret

Corrective and Preventative Measures

- **API**: stop attaching tokens to email links, keeping that for Sign In / Sign Up flow.

Why we were lucky?

- The issue was responsively disclosed to us by a trusted user.
- The bug would only affect admins of the same collective/host, people who likely know each other and would not harm themselves.

What went well?

- Fast CI and deployment. Fast time to ship.
- Ability to invalidate tokens by rotating JWT secret.
- Responsive support.

0 comments on commit b3f0a99

Please sign in to comment.
You can’t perform that action at this time.