Add Incident Report for 2019-02-20 Versionista Scraper Silent Failure #17
Conversation
Versionista imports started silently failing on 2019-02-12 and we didn't detect it until a user reported some unusual logs a week later.
| ### What Went Wrong | ||
|
|
||
| - This didn’t register as an actual error and we had no alerts about it for a week. | ||
| - Other people saw the issue before @Mr0grog but did not respond or investigate or flag it for someone who could. |
There was a problem hiding this comment.
Who (besides @jjudish) saw this before you? Was this an analyst/dev communication gap?
There was a problem hiding this comment.
I know at least @lightandluck did, because the reason I saw the issue as early as I did was his comment on Slack. But he did not note that there was a potential major data problem: https://archivers.slack.com/archives/C47DJK8H0/p1550710886000900
(That’s actually why the first second entry in the timeline is me replying to the issue from my phone and noting I’d take a look when I got to a computer.)
There was a problem hiding this comment.
Yea, sorry, I really didn't understand how big of a problem it was! I assumed it was just a problem on their end because I thought we had things covered by Sentry. I didn't think troubleshooting someone else's instance would be urgent. But know now that if someone is having problems with our software, we might have the same problem! Will remember in the future.
(Really I was dumbstruck by the fact that someone else was using the scraper for a year and we had never heard anything from them. But finding out it was WIP was anti-climatic)
There was a problem hiding this comment.
Man, this makes me really want to go read all the other incident reports. Good lesson learned. Thanks y'all for your diligence 👍 💯 💪
| ### What Went Wrong | ||
|
|
||
| - This didn’t register as an actual error and we had no alerts about it for a week. | ||
| - Other people saw the issue before @Mr0grog but did not respond or investigate or flag it for someone who could. |
jsnshrmn
deleted the
incident-2019-02-20--versionista-scraper-silent-failure
branch
Versionista imports started silently failing on 2019-02-12 and we didn't detect it until a user reported some unusual logs a week later.
The fix for this lives in edgi-govdata-archiving/web-monitoring-versionista-scraper#196, which has not yet been reviewed and merged, but is currently running in production.