Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Change threads field behaviour #399

Closed
Jeffail opened this issue Mar 7, 2020 · 1 comment
Closed

Change threads field behaviour #399

Jeffail opened this issue Mar 7, 2020 · 1 comment
Labels
enhancement processors Any tasks or issues relating specifically to processors v4 An issue proposed for v4 of Benthos

Comments

@Jeffail
Copy link
Collaborator

Jeffail commented Mar 7, 2020

Currently the threads field of pipeline is used to create an explicit number of processing threads. This can be useful as sometimes it's desired to limit the workload of a pipeline to one single logical thread (conditional sleeping), and sometimes it's useful to spawn a much larger number of threads than CPU cores if the workload is mostly IO bound (these are green threads anyway).

However, I suspect that for the vast majority of users the best behaviour for them is to have as many processing threads as there are CPU cores. In this case our defaults are failing as one pipeline is almost never going to be the ideal number.

I'd like to keep this value as a configured field because that allows you to have different thread counts for isolated streams running in streams mode. Therefore I propose changing to default to 0 and documenting that a value of 0 means Benthos matches the number of logical CPU threads on the running machine.

Since this would be a breaking change for some hypothetical configs I'll wait for Benthos v4 to implement this.

@Jeffail Jeffail added enhancement processors Any tasks or issues relating specifically to processors v4 An issue proposed for v4 of Benthos labels Mar 7, 2020
@Jeffail
Copy link
Collaborator Author

Jeffail commented Feb 25, 2022

Done da1b9d8

@Jeffail Jeffail closed this as completed Feb 25, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement processors Any tasks or issues relating specifically to processors v4 An issue proposed for v4 of Benthos
Projects
None yet
Development

No branches or pull requests

1 participant