Skip to content

Commit

Permalink
HSEARCH-3879 Clarify the indexing queue settings
Browse files Browse the repository at this point in the history
  • Loading branch information
yrodiere authored and fax4ever committed Apr 3, 2020
1 parent 841a9ef commit 33365e0
Show file tree
Hide file tree
Showing 2 changed files with 15 additions and 7 deletions.
17 changes: 11 additions & 6 deletions documentation/src/main/asciidoc/backend-elasticsearch.asciidoc
Expand Up @@ -799,20 +799,25 @@ hibernate.search.backends.<backend name>.index_defaults.indexing.max_bulk_size 1
* `indexing.queue_count` defines the number of queues.
Expects a strictly positive integer value.
+
Higher values may lead to higher indexing throughput,
but incur a risk of <<backend-elasticsearch-indexing-queues-circuit-breaker,overloading Elasticsearch>>.
Higher values will lead to more connections being used in parallel,
which may lead to higher indexing throughput,
but incurs a risk of <<backend-elasticsearch-indexing-queues-circuit-breaker,overloading Elasticsearch>>,
leading to Elasticsearch giving up on some requests and resulting in indexing failures.
* `indexing.queue_size` defines the maximum number of elements each queue can hold.
Expects a strictly positive integer value.
+
Lower values may lead to lower memory usage, especially if there are many queues,
but values that are too low will reduce the likeliness of reaching the max bulk size
and increase the likeliness of <<backend-elasticsearch-indexing-queues-blocking,application threads blocking>>,
potentially leading to lower indexing throughput.
and increase the likeliness of <<backend-elasticsearch-indexing-queues-blocking,application threads blocking>>
because the queue is full,
which may lead to lower indexing throughput.
* `indexing.max_bulk_size` defines the maximum number of indexing requests in each bulk request.
Expects a strictly positive integer value.
+
Higher values may lead to higher indexing throughput,
but incur a risk of <<backend-elasticsearch-indexing-queues-circuit-breaker,overloading Elasticsearch>>.
Higher values will lead to more documents being sent in each HTTP request sent to Elasticsearch,
which may lead to higher indexing throughput,
but incurs a risk of <<backend-elasticsearch-indexing-queues-circuit-breaker,overloading Elasticsearch>>,
leading to Elasticsearch giving up on some requests and resulting in indexing failures.
+
Note that raising this number above the queue size has no effect,
as bulks cannot include more requests than are contained in the queue.
Expand Down
5 changes: 4 additions & 1 deletion documentation/src/main/asciidoc/backend-lucene.asciidoc
Expand Up @@ -603,7 +603,8 @@ hibernate.search.backends.<backend name>.index_defaults.indexing.queue_size 1000
* `indexing.queue_count` defines the number of queues.
Expects a strictly positive integer value.
+
Higher values may lead to higher indexing throughput if CPU is the bottleneck when indexing.
Higher values will lead to more indexing operations being performed in parallel,
which may lead to higher indexing throughput if CPU power is the bottleneck when indexing.
+
Note that raising this number above the <<backend-lucene-threads,number of threads>> is never useful,
as the number of threads limits how many queues can be processed in parallel.
Expand All @@ -613,6 +614,8 @@ Expects a strictly positive integer value.
Lower values may lead to lower memory usage, especially if there are many queues,
but values that are too low will increase the likeliness of
<<backend-lucene-indexing-queues-blocking,application threads blocking>>
because the queue is full,
which may lead to lower indexing throughput.

[TIP]
[[backend-lucene-indexing-queues-blocking]]
Expand Down

0 comments on commit 33365e0

Please sign in to comment.