Introduce Optional Cap on Number Of Threads Per Query [LUCENE-8808]

[asfimport]

With the presence of #9802 , a natural progression is to allow advanced users to specify a cap on the number of threads a query can use. This is especially useful for long polled IndexSearcher instances, where the same instance is used to fire queries for multiple runs and there are multiple concurrent IndexSearcher instances on the same node.

 

This will be an optional parameter and local only to the IndexSearcher instance being configured during construction.

---

Migrated from LUCENE-8808 by Atri Sharma (@atris), updated May 24 2019

[asfimport]

Atri Sharma (@atris) (migrated from JIRA)

@jpountz Does this seem like a useful thing?

[asfimport]

Adrien Grand (@jpountz) (migrated from JIRA)

Can you explain the use-case a bit more? What does it buy compared to having an executor with a fixed number of threads that is shared across all IndexSearcher instances?

[asfimport]

Atri Sharma (@atris) (migrated from JIRA)

@jpountz If a shared executor is used across all IndexSearchers, then there is a probability that a few odd large queries will hog all available threads on the executor, and the remaining queries can be queued in the executor's queue. This parameter is an effort to ensure that a small subset of queries does not overload the shared resource.

[asfimport]

Adrien Grand (@jpountz) (migrated from JIRA)

I don't think it really fixes the problem, it only makes it occur a bit later? If there is a single bad query, it makes the issue less problematic for the well-behaved queries, but what if 10% of your queries are bad, then these queries would likely still hog all available threads (and probably even fill the queue) and none of the well-behaved queries may run?

[asfimport]

Atri Sharma (@atris) (migrated from JIRA)

@jpountz Let me elaborate by taking an example:

 

Consider a scenario where the underlying hardware is a 16 vCPU machine. There is a  single large index (approx 450 GB).

Now, N concurrent IndexSearchers sharing the same executor are firing different queries. This could mean that some queries could have X number of slices, where X >> 16. Other queries could need number of slices << 16.

 

If we had a hard cap of the number of slices a single query can have, that will mean that the large query will be bound by the N.

I agree that if there is a sizeable percentage of the overall query set which will ideally need slices >> than the number of cores present, then eventually the executor's queue will build up. However, that seems more like a problem of hardware underprovisioning than a limitation in Lucene?

 

The essential idea is to have a best-effort way to limit the amount of resources a single query can use. As you highlighted, this will not cover all edge cases, but should be better than nothing. WDYT?

[asfimport]

Atri Sharma (@atris) (migrated from JIRA)

@jpountz I further tested this by trying out a 900 GB dataset with a large query (around 6 billion hits) on a 32 vCPU machine. With 8 concurrent clients doing the same query and sharing the same threadpool, the backlog of threads on the threadpool was 3-5X larger than the case when there was a capping of 16 threads per query. Overall, the P50 improved with the cap since the extra dequeuing and thread context switching costs were saved. However, P99.9 worsened due to lesser concurrency.

 

Since this is an optional parameter, users will have a choice, so is configurable per workload.

 

WDYT?

[asfimport]

Adrien Grand (@jpountz) (migrated from JIRA)

In general I'm reluctant to adding more configuration options. It increases the API surface area, which in turn makes Lucene harder to use. My gut feeling right now is that this new setting would not help enough to warrant the introduction of a new option?

[asfimport]

Atri Sharma (@atris) (migrated from JIRA)

Sure. Do you have any ideas of how could we control that large queries do not go out of hand in terms of resource consumption? Happy to discuss them

[asfimport]

Adrien Grand (@jpountz) (migrated from JIRA)

Sorry I don't. This is a hard problem. Your approach makes the assumption that it's ok to make slow queries slower if that helps fast queries remain fast, which I don't think would be the right assumption for most users?

[asfimport]

Atri Sharma (@atris) (migrated from JIRA)

@jpountz Essentially, no. My proposal is basically to have a last level limit to ensure that no single query overloads the entire system i.e. a query does not decide to use 72 threads on a 32 core machine. We could potentially not even have a configurable cap, and always set it to number of cores.

 

I am not married to this approach, so happy to hear ideas.