Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a circuit breaker to the size of the state object in Task Manager tasks #182186

Open
4 tasks
mikecote opened this issue Apr 30, 2024 · 1 comment
Open
4 tasks
Labels
Feature:Task Manager Team:ResponseOps Label for the ResponseOps team (formerly the Cases and Alerting teams)

Comments

@mikecote
Copy link
Contributor

Problem Statement

We have no guardrails when it comes to storing state within a Task Manager task, allowing some implementations to abuse what is stored within the state, causing the Task Manager performance to degrade by serializing / deserializing large amounts of data during the claiming phase.

We should start by logging a warning when we save a task with a state larger than 1 MB. This will let us collect how frequently this is happening.

Definition of Done

  • Warning message is logged when a task finishes running and has a state larger than 1 MB
  • Warning message is logged when a task is scheduled and has a state larger than 1 MB
  • Warning message includes the configured limit (1 MB)
  • Warning message include the observed size (X MB)
@mikecote mikecote added Feature:Task Manager Team:ResponseOps Label for the ResponseOps team (formerly the Cases and Alerting teams) labels Apr 30, 2024
@elasticmachine
Copy link
Contributor

Pinging @elastic/response-ops (Team:ResponseOps)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Feature:Task Manager Team:ResponseOps Label for the ResponseOps team (formerly the Cases and Alerting teams)
Projects
None yet
Development

No branches or pull requests

2 participants