Skip to content

Conversation

@tw4l
Copy link
Member

@tw4l tw4l commented Nov 8, 2023

Fixes #1344

Sends email to superadmin when a background job fails.

Sample email (from operator logs, from job that failed because of invalid replica s3 bucket secret access key):

Screen Shot 2023-11-07 at 11 29 23 PM

tw4l added 3 commits November 7, 2023 23:04
When we add additional types of background jobs that don't have
the same fields, will have to do this a bit differently. For now
we can assume all jobs have the same fields.
@tw4l tw4l requested a review from ikreymer November 8, 2023 04:24
@tw4l tw4l changed the title Issue 1344 failed background job alert Send email to superuser when background job fails Nov 8, 2023
)
superuser = await self.user_manager.get_superuser()
org = await self.org_ops.get_org_by_id(job.oid)
self.email.send_background_job_failed(job, org, finished, superuser.email)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just in case, should wrap this in run_in_executor() so it doesn't block the async event loop handling for the operator.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now added!

@ikreymer ikreymer merged commit 30bbefb into main Nov 9, 2023
@ikreymer ikreymer deleted the issue-1344-failed-background-job-alert branch November 9, 2023 03:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add mechanism to inform superadmin when replication or other background jobs fail

3 participants