Improving CSV generation by supporting concurrent background tasks #181064
Labels
Feature:Reporting
Reporting (PDF, CSV, ..) feature
Team:SharedUX
Team label for AppEx-SharedUX (formerly Global Experience)
The
report:execute
task today has a concurrency per Kibana node set to1
. What Task Manager does when a task type is configured like this is it will prevent more than one reporting task from running on the same node at any given time.The following exposes some limitations that we have in serverless:
What I propose is running the CSV generation tasks under a new task type
report:execute-csv
that doesn't havemaxConcurrency
set within its task definition and keep thereport:execute
multi-purpose in case there are still CSV tasks in the queue. This will allow 10x throughput per Kibana node for generating CSVs and will benefit serverless, ESS and on-prem users. One thing to keep an eye out for is with 10x concurrency, we also put 10x the memory / CPU pressure and I am not familiar with the internals of how much resource utilization each task needs.The text was updated successfully, but these errors were encountered: