Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Better share threadpools #12666

Closed
jpountz opened this issue Aug 5, 2015 · 1 comment
Closed

Better share threadpools #12666

jpountz opened this issue Aug 5, 2015 · 1 comment
Labels
>bug discuss

Comments

@jpountz
Copy link
Contributor

jpountz commented Aug 5, 2015

We have 17 threadpools today, and while all of them are configured individually to create a number of threads that is reasonable for the number of cores, summing up all created threads gives a high number. For instance a machine with 4 cores starts between 30 (only considering fixed threadpools) and 59 threads (considering fixed + scaling threadpools), and a machine with 32 cores starts between 215 and 373 threads. And this doesn't account for threads that are created by Lucene itself.

These numbers look too high to me, and while we could decrease the size of individual thread pools, this would have the downside that eg. only issuing get requests could not make use of a whole machine's resources. So maybe we could look at better sharing threadpools across tasks? For instance I was thinking we could merge the following threadpools:

  • get, search, suggest, percolate, warmer (read operations)
  • index, bulk, refresh, flush (write operations)
  • fetch_shard_started, fetch_shard_stored
jpountz added a commit to jpountz/elasticsearch that referenced this issue Aug 17, 2015
@jpountz
Share thread pools that have similar purposes.
fbdf230 
Because we have thread pools for almost everything, even if each of them has a
reasonable size, the total number of threads that elasticsearch creates is
high-ish. For instance, with 8 processors, elasticsearch creates between 58
(only fixed thread pools) and 111 threads (including fixed and scaling pools).
With this change, the numbers go down to 33/59.

Ideally the SEARCH and GET thread pools should be the same, but I couldn't do
it now given that some SEARCH requests block on GET requests in order to
retrieve indexed scripts or geo shapes. So they are still separate pools for
now.

However, the INDEX, BULK, REFRESH and FLUSH thread pools have been merged into
a single WRITE thread pool, the SEARCH, PERCOLATE and SUGGEST have been merged
into a single READ thread pool and FETCH_SHARD_STARTED and FETCH_SHARD_STORE
have been merged into FETCH_SHARD. Also the WARMER pool has been removed: it
was useful to parallelize fielddata loading but now that we have doc values by
default, we can make things simpler by just loading them in the current thread.

Close elastic#12666
@jpountz jpountz mentioned this issue Aug 17, 2015
Closed
@jpountz jpountz mentioned this issue Oct 21, 2015
Closed
@jpountz
Copy link
Contributor Author

jpountz commented Jan 26, 2016

No consensus could be reached. Closing.

@jpountz jpountz closed this as completed Jan 26, 2016
@clintongormley clintongormley mentioned this issue Apr 29, 2016
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
>bug discuss
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Share thread pools that have similar purposes. jpountz/elasticsearch
2 participants
@clintongormley @jpountz