[Task Manager] Evenly distribute bulk-enabled alerting rules

[Zacqary]

Summary

Closes Part of #171980

When bulkEnableing more than 1 task, adds a random delay to each subsequent task's runAt and scheduledAt to more evenly distribute their execution times. This offset is a maximum of 5 minutes, or the task's interval, whichever is shorter.

As per Slack discussion with @mikecote, this is a random distribution of execution times instead of a predictable, algorithmic offset. We believe that a random distribution will do a better job of avoiding spikes than anything more directed.

Checklist

• Unit or functional tests were updated or added to match the most common scenarios

[apmmachine]

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

• /oblt-deploy : Deploy a Kibana instance using the Observability test environments.
• /oblt-deploy-serverless : Deploy a serverless Kibana instance using the Observability test environments.
• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[elasticmachine]

Pinging @elastic/response-ops (Team:ResponseOps)

[kibana-ci]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: 5cd0021

Failed CI Steps

• Rule Management - Prebuilt Rules - Security Solution Cypress Tests #3
• Investigations - Security Solution Cypress Tests #7

Test Failures

• [job] [logs] Investigations - Security Solution Cypress Tests #7 / [ESS] Save Timeline Prompts "before each" hook for "should NOT prompt when navigating with a changed and unsaved timeline from the page where timeline is disabled" "before each" hook for "should NOT prompt when navigating with a changed and unsaved timeline from the page where timeline is disabled"

Metrics [docs]

✅ unchanged

History

• 💛 Build #182316 was flaky 067b2cf

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

[mikecote]

I learned a few things as I started reviewing this PR, we'll need to address these before we benefit from this change:

• Security solution bulk enable still calls the alerting enable API , one rule at a time (this prevents the bulkEnable flow to be utilized)
• After using our bulkEnableRules API, the underlying code currently schedules the underlying tasks one-by-one whenever it's missing (including the flow of installing then enabling the prebuild security rules)
• The bulk enable API floods Elasticsearch with requests trying to migrate legacy actions [ERROR][plugins.alerting] migrateLegacyActions(): Failed to migrate legacy actions for SIEM rule c390e56e-c493-48a8-8d9c-b67ae0f7a072: connect ECONNRESET 127.0.0.1:9200 - Local: unknown:unknown, Remote: unknown:unknown

Happy to brainstorm if needed.

cc @kobelb

[Zacqary]

@mikecote Should we merge this PR but change the definition of done on the issue? I feel like this change is at least part of the solution.

[mikecote]

@mikecote Should we merge this PR but change the definition of done on the issue? I feel like this change is at least part of the solution.

Yes we can continue with this PR as part of the solution. I'll continue my review in it's current state but I think we'll need to hold off closing the #171980 issue until we fix the last two bullet points that I mentioned earlier, and we'll need to ask or work with Security Solution to move to use our bulk enable API. cc @XavierM

I think the main goal @kobelb wants solved is when enabling > 1,000 security solution rules that we don't make the K8s autoscaler go crazy. I think it would need the bullet points I mentioned earlier to be solved.

[mikecote]

Changes LGTM! Tested locally by creating some rules in stack management and using the bulk enable / disable API to see the calculations apply on enable.