Switch Reporting to Task Manager

[tsullivan]

Summary

Remove the ES-Queue sub-service of Reporting and replace the functionality with Task Manager integration.

Part of #53900

Closes #69340
Closes #75603
Closes #75605
Closes #76411
Closes #87074

Blocked on:

• Refactoring:
  • [Reporting] Update more Server Types for TaskManager #74915
  • [Reporting] rename interfaces to align with task manager integration #76716
  • [Reporting] Clean Up TypeScript Definitions #76566
• [Task Manager] Support for limited concurrency Task Types #54916
• Core should provide a synchronous API for reading config values #82723

Summary

Using Task Manager for Reporting can be done by registering 2 new task types:

• Execute Report Task
• Monitor Reports Task

Execute Report Task
The Execute Report task is a "one-off" task that Task Manager will trigger as soon as it is executed. The Reporting API that handles requests to generate new reports will be responsible for scheduling these tasks, and also storing a historical document for the report output in the .reporting-* indices. The scheduled task references this placeholder report document.\

This design splits report execution into 2 phases for async report generation:

• The schedule task phase uses logic migrated from the ESQueue "create job" phase, and calls the CreateJobFn of an Export Type.
  • Enrich the parameters provided from the Generation API into job parameters for an export type.
  • Store a historical document for a "Pending" report job.
  • New: schedule a task with Task Manager to asynchronously run the next phase
• The run task phase uses logic migrated from the the "run task" phase in ESQueue, and calls the RunTaskFn of an Export type.
  • set the status of the historical document is set from "Pending" to "Processing."
  • generate the report output
  • save the output into the historical document, and set the status to "Completed"

---

Generate Report: Create Job Phase:

---

Generate Report: Run Task Phase:

---

Monitor for Pending Reports Task
It is possible for a task in the run task phase to fail without rescheduling itself. This will happen in situations such as:

• The Kibana server restarts in the middle of running an Execute Report task
• Report jobs were queued with ESQueue before a stack upgrade, and still have "Pending" status.
• Unexpected situations (server crashes, task data gets deleted accidentally, etc)

ESQueue was able to self-heal against this problem since it used the .reporting-* as the queue index and the historical storage index. That means "stuck" jobs pending would be discovered in the historical data in the same cycle ESQueue uses to find queued work that needs to be claimed. ESQueue would immediately claim the stuck/expired jobs it found.

In Task Manager, we will need to define a 2nd task that discovers stuck jobs, by running searches on "Pending" or "Processing" jobs in the historical data and filtering on the expiration_time field. Unlike ESQueue, it will not immediately claim these jobs once discovered; instead it will re-schedule the execute report task using the job parameters in the historical document.

---

Monitor for Pending Reports:

---

Checklist

Delete any items that are not applicable to this PR.

• Unit or functional tests were updated or added to match the most common scenarios

For maintainers

• This was checked for breaking API changes and was labeled appropriately

Release Note for breaking change

Make sure there are no pending reporting jobs before upgrading to 8.0. If there are any, they will remain in pending state indefinitely.

[gmmorris]

I looked at the TM code and it looks right.
I started Kibana and saw that the Execute Report task did in fact have the limited concurrency expected.

I haven't tested that reports are generated or anything along those lines.

[tsullivan]

@streamich shared logs that appeared as though reporting jobs are running twice, so I'm holding off on this for a bit while I investigate.

[tsullivan]

@elasticmachine merge upstream

[streamich]

LGTM, tested on Mac/Chrome, everything seems to work, could not reproduce the errors I was seeing a few weeks ago.

[tsullivan]

This PR adds a breaking change: if you have pending reports in the cluster, they will not be continued after upgrading to a version with these changes.

Because of this fact, @kobelb and I have chatted and decided it's best for this change to roll out in a major version. Therefore this PR will not be backported to 7.x.

[kibanamachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: 2b7b49c
• Storybooks Preview
• Documentation Changes

Metrics [docs]

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
reporting	95.9KB	96.2KB	+328.0B

History

• 💚 Build #111621 succeeded 2b7b49c
• 💚 Build #111600 succeeded 8dbabd5
• 💔 Build #111569 failed 65d752d
• 💔 Build #111523 failed 907d209
• 💔 Build #111488 failed aab4890

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream