feat(worker): Add support for concurrent pollers

[mjameswh]

What changed

• Added WorkerOptions properties maxConcurrentWorkflowTaskPolls and maxConcurrentActivityTaskPolls, which allow controlling the number of pollers to be used to fetch Workflow/Activity tasks from the Task Queue. Properly adjusting these values should allow better filling of the corresponding execution slots.
• Remove the @experimental tag on reuseV8Context
• Better auto-configuration of maxCachedWorkflows, including using a different formula when reuseV8Context is enabled (fix [Bug] Default WorkerOptions.maxCachedWorkflows is too low #838)
• Default value for maxConcurrentWorkflowTaskExecutions has been reduced to 40 (was previously 100), as higher values increase the risk of Workflow Task Timeouts unless other options are also tuned. This was not problem previously because the single poller was unlikely to fill in all execution slot anyway, so max would rarely reached).
• Load balance create workflows requests amongst WF threads (fix [Feature Request] Better balancing of workflows between worker threads #608)
• Better documentation of some performance/scaling-related properties in WorkerOptions.
• Add more options to internal performance test runners, including specifying mTLS options (for testing against Temporal Cloud, pollers and WF thread pool size)
• Updated Core to 782282e7

[mjameswh]

@bergundy Some major changes from the previous pass. Please take another look..

@Sushisource Would you mind taking a second look at changes in the worker-options.ts file? Any runtime constraint I forgot to check?

Also, in tests against Temporal Cloud, I had to set both WF and Act pollers to ~60 each to get peak performances for a single thread Worker running on my laptop. That seems way too high to use as a default value, but performance with only 2 pollers (ie. the minimum) were very disappointing. I settled on 10 pollers WF/Act as the default value. Comments?

[Sushisource]

I think the option defaults make sense 👍

Just some wording fixes in the docstrings.

[mjameswh]

I think the option defaults make sense 👍

Just some wording fixes in the docstrings.

Thanks a lot! I need to find some good English spell/grammar checker that knows how to deal with source code, but hopefully, in the mean time, there is peer reviews 😆

[bergundy]

But it may have a negative impact on the cluster's perf and may result in workflow tasks timing out if they're not processed in a timely fashion.

[mjameswh]

As far as I could see in perf tests, task timeouts were linked to setting maxConcurrentWorkflowTaskExecutions too high, not to increasing maxConcurrentWorkflowTaskPolls. Core won't poll anyway if there is no execution slot available.

Still, I will change the phrasing to mention that too high might be negative on the server cluster.

[bergundy]

I'm wondering what the CPU load was and number of wf/s (not that we need to document that).

[bergundy]

Awesome work on this!