[QUESTION]: Can you increase the number of HTTP tasks picked up by the workers?

[headibr]

Is there somewhere in the configuration setting to define the number of HTTP tasks picked up by the workers during each poll cycle?

[FpyrypT]

Hello! I suggest that such property doesn't exist in java client. I think it's possible to write a similar client https://github.com/Netflix/conductor/blob/main/client/src/main/java/com/netflix/conductor/client/automator/TaskPollExecutor.java#L106
with this method
https://github.com/Netflix/conductor/blob/main/client/src/main/java/com/netflix/conductor/client/http/TaskClient.java#L167.
Also, TaskPollExecutor can be rewritten to batch polling with an underlying thread pool for tasks execution.

[aravindanr]

@headibr The poll count for system task workers (HTTP) is controlled by this property, conductor.app.system-task-max-poll-count. The default value is 1.

[headibr]

@headibr The poll count for system task workers (HTTP) is controlled by this property, conductor.app.system-task-max-poll-count. The default value is 1.

@aravindanr Are there any consequences by increasing that value to 500 or even 1000? My system consistently has 25k-30k running workflows at any given time but HTTP tasks slow down when the queue builds up.

[headibr]

@aravindanr in which config file is this property located? or is this something we need to manually add?

[jxu-nflx]

@headibr You can add it manually to your config file

[aravindanr]

@headibr Increasing the count to a large value does not speed up processing. The value should be a function of the number of processors (e.g. no of processors * 4) on the server that Conductor runs on. Setting a high value will simply overload the system with many threads. To reduce task build up, you can scale horizontally by adding more servers.