Add incident report: 2023-02-01 leap hub, dask-gateway induced outage #4
Conversation
| @@ -0,0 +1,20 @@ | |||
| # 2023-02-01 Heavy use of dask-gateway induced critical pod evictions | |||
|
|
|||
There was a problem hiding this comment.
Please add summary statement and impact.
|
|
||
| All times in UTC+1 | ||
|
|
||
| - 2023-02-01 - [Summary of issue updated between ~8-9 PM](https://github.com/2i2c-org/infrastructure/issues/2126#issuecomment-1412554908) |
There was a problem hiding this comment.
Generally I'd expect to see a full timeline here not a link - eg https://sre.google/sre-book/example-postmortem/
However looking at the other incident reports in this repo it doesn't seem like we have a clear shared expectation of what should be here.
|
|
||
| ## What went wrong | ||
|
|
||
| - I believe various critical pods on core nodes pods got evicted when prometheus started scraping from ~200 nodes metrics exporters |
There was a problem hiding this comment.
Can we do better than belief? Is there not evidence to support your hypothesis here? That's what I'd expect the incident report/postmortem to reveal through analysis of the contributing factors/root causes/triggers.
|
|
||
| ## Follow-up improvements | ||
|
|
||
| - #2127 |
There was a problem hiding this comment.
Please make these full markdown links with a summary line
| ## What went wrong | ||
|
|
||
| - I believe various critical pods on core nodes pods got evicted when prometheus started scraping from ~200 nodes metrics exporters | ||
| - I think its likely, but I can't say for sure, that the dask scheduler pod also would run into resource limitations with this amount of workers |
There was a problem hiding this comment.
Is this what went wrong? I'm not sure what this speculation is for in terms of analyzing the incident. Is this a lesson learned?
|
@pnasrat thank you for the review! These are all relevant points, I'll leave them unanswered for now since I'm feeling a time pressure to focus on other things. |
Marked as draft pending time available to refine this further based on feedback!
Related
leapclustersprodhub - massive node scale up, hub pod restarted for unknown reason infrastructure#2126