Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Send (OOM) kills to Sentry #452

Closed
MrSerth opened this issue Sep 17, 2023 · 3 comments · Fixed by #477
Closed

Send (OOM) kills to Sentry #452

MrSerth opened this issue Sep 17, 2023 · 3 comments · Fixed by #477
Labels
deployment Everything related to our production environment enhancement New feature or request

Comments

@MrSerth
Copy link
Member

MrSerth commented Sep 17, 2023

Currently, when Poseidon is killed, e.g., by the OOM killer, we do not get notified nor any Sentry event. Ideally, we would add some hook or other mechanism to get informed whenever Poseidon quits unexpectedly.

@MrSerth MrSerth added enhancement New feature or request deployment Everything related to our production environment labels Sep 17, 2023
@MrSerth
Copy link
Member Author

MrSerth commented Sep 25, 2023

In order to address this issue, we want to tackle two different aspects:

  • First, we want to add an ExecStopPost, checking the exit code of Poseidon and potentially creating a Sentry issue.
  • Second, we want to formalize our memory monitoring introduced with Introduce debug memory monitoring #457, so that the threshold is customizable and can remain in the source code. This should allow enabling / disabling the monitoring at all. If a memory leak is detected, the information is written as an error to the log and sent to Sentry (if enabled).

@mpass99
Copy link
Contributor

mpass99 commented Oct 6, 2023

Should we also address setting up the systemd watchdog within this issue (or create a separate issue for it)?

@MrSerth
Copy link
Member Author

MrSerth commented Oct 6, 2023

Let's extract it to a dedicated issue, since it is not directly related to the OOM kills tracked in this one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
deployment Everything related to our production environment enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants