Task manager enhancements for error handling in alerting and actions #39829
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…k started running
|
Pinging @elastic/kibana-stack-services |
mikecote
added
the
release_note:skip
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
|
I didn't do a full review, but @mikecote asked me to have a look at the migration changes, and those look peachy. 💯 |
|
Hi, sorry I need a little more time looking at this. I pulled it down, and getting a server crash, but there's a chance it's because my ES is messed up or other reasons. Also, there are 2 conflicts that need to be resolved on this PR |
|
I think I keep hitting stuck migrations, even when I clear all my documents out. I can fix the problems with missing shards in the indices (hence why they are red) but for now, this is causing Kibana to completely crash. I'll need to look a little more to see if this is just a distraction |
This comment has been minimized.
This comment has been minimized.
tsullivan
approved these changes
Jul 22, 2019
joshdover
reviewed
Jul 23, 2019
|
I'd feel way more comfortable if the SavedObjects/Migration changes were split into a separate PR from this one. |
|
@joshdover I have moved it to another PR #41815, will update this one once merged. 👍 |
This comment has been minimized.
This comment has been minimized.
|
@joshdover The migration changes have been removed from this PR now that they are in master. Let me know if this looks good to you now. |
💚 Build Succeeded |
joshdover
approved these changes
Jul 25, 2019
mikecote
added a commit
to mikecote/kibana
that referenced
this pull request
Jul 25, 2019
…lastic#39829) * Allow mtask definitions to overwrite default setting maxAttemps * Leverage scheduledAt from task manager * Treat maxAttempts like attempts and not retries * Add support for second intervals * Min 1 attempt * Reverse relying on scheduledAt * Add new startedAt attribute in task manager that keeps track when task started running * Don't extend runAt when claiming a task * Remove startedAt from state * Attempt trying to define custom getBackpressureDelay function * Pass error object to getBackpressureDelay * Cleanup processResultForRecurringTask code * Add backpressure to timed out tasks * Change default timeout backpressure calculation * getBackpressureDelay to return seconds instead of milliseconds * Add comment for task store query * Compress query * Revert alert / actions specific code * Add more interval tests * Fix failing jest tests * Fix test * Add more unit tests * Fix integration tests * Fix sorting of tasks to process * WIP * Always provide error when getBackpressureDelay is called * Rename getBackpressureDelay to getRetryDelay * retryAt to be calculated from timeout time by default * Remove invalid test * Add unit tests * Consider timeout before scheduling a retryAt * Remove backpressure terminology * Remove support for 0 based intervals and timeouts * Apply PR feedback * Fix last place using Math.abs * Modify migrations to allow running a script when converting an index to an alias * Convert task manager to use saved objects * Fix broken test * Fix broken tests pt1 * Remove index from task manager config schema * Accept platform changes * PR feedback * Apply PR feedback * Apply PR feedback pt2 * Apply PR feedback pt3 * Apply PR feedback pt4 * Fix feedback pt3 * Rename RawSavedObjectDoc to SavedObjectsRawDoc
mikecote
added a commit
to mikecote/kibana
that referenced
this pull request
Jul 26, 2019
…lastic#39829) * Allow mtask definitions to overwrite default setting maxAttemps * Leverage scheduledAt from task manager * Treat maxAttempts like attempts and not retries * Add support for second intervals * Min 1 attempt * Reverse relying on scheduledAt * Add new startedAt attribute in task manager that keeps track when task started running * Don't extend runAt when claiming a task * Remove startedAt from state * Attempt trying to define custom getBackpressureDelay function * Pass error object to getBackpressureDelay * Cleanup processResultForRecurringTask code * Add backpressure to timed out tasks * Change default timeout backpressure calculation * getBackpressureDelay to return seconds instead of milliseconds * Add comment for task store query * Compress query * Revert alert / actions specific code * Add more interval tests * Fix failing jest tests * Fix test * Add more unit tests * Fix integration tests * Fix sorting of tasks to process * WIP * Always provide error when getBackpressureDelay is called * Rename getBackpressureDelay to getRetryDelay * retryAt to be calculated from timeout time by default * Remove invalid test * Add unit tests * Consider timeout before scheduling a retryAt * Remove backpressure terminology * Remove support for 0 based intervals and timeouts * Apply PR feedback * Fix last place using Math.abs * Modify migrations to allow running a script when converting an index to an alias * Convert task manager to use saved objects * Fix broken test * Fix broken tests pt1 * Remove index from task manager config schema * Accept platform changes * PR feedback * Apply PR feedback * Apply PR feedback pt2 * Apply PR feedback pt3 * Apply PR feedback pt4 * Fix feedback pt3 * Rename RawSavedObjectDoc to SavedObjectsRawDoc
mikecote
added a commit
that referenced
this pull request
Jul 26, 2019
…39829) (#42004) * Allow mtask definitions to overwrite default setting maxAttemps * Leverage scheduledAt from task manager * Treat maxAttempts like attempts and not retries * Add support for second intervals * Min 1 attempt * Reverse relying on scheduledAt * Add new startedAt attribute in task manager that keeps track when task started running * Don't extend runAt when claiming a task * Remove startedAt from state * Attempt trying to define custom getBackpressureDelay function * Pass error object to getBackpressureDelay * Cleanup processResultForRecurringTask code * Add backpressure to timed out tasks * Change default timeout backpressure calculation * getBackpressureDelay to return seconds instead of milliseconds * Add comment for task store query * Compress query * Revert alert / actions specific code * Add more interval tests * Fix failing jest tests * Fix test * Add more unit tests * Fix integration tests * Fix sorting of tasks to process * WIP * Always provide error when getBackpressureDelay is called * Rename getBackpressureDelay to getRetryDelay * retryAt to be calculated from timeout time by default * Remove invalid test * Add unit tests * Consider timeout before scheduling a retryAt * Remove backpressure terminology * Remove support for 0 based intervals and timeouts * Apply PR feedback * Fix last place using Math.abs * Modify migrations to allow running a script when converting an index to an alias * Convert task manager to use saved objects * Fix broken test * Fix broken tests pt1 * Remove index from task manager config schema * Accept platform changes * PR feedback * Apply PR feedback * Apply PR feedback pt2 * Apply PR feedback pt3 * Apply PR feedback pt4 * Fix feedback pt3 * Rename RawSavedObjectDoc to SavedObjectsRawDoc
mikecote
moved this from Done for 7.4
to Done (Ordered by most recent)
in Make it Action
1 task
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In this PR, we're making the following changes to the task manager to support having better error handling for alerting and actions:
10s)10s)maxAttemptsconfigurationmaxAttemptshas a minimum of 1getRetryDelayfunction to return delay in seconds to wait before attempting the task againmaxAttemptsconfiguration now behaves like maximum total attempts and not how many retries after first failurerunAtis no longer extended by timeout when the task runsFixes #40872
Dev-Docs
Task manager now uses saved objects
Starting up kibana will convert
.kibana_task_managerto an alias and indices will follow the.kibana_task_manager_1syntax. A migration will execute to prefix the ids withtask:as it converts to saved objects.