-
Notifications
You must be signed in to change notification settings - Fork 24.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Very large terms query takes a long time to return a validation error #67031
Comments
Pinging @elastic/es-search (Team:Search) |
These results are expected with the current design. Today we validate the We could also make the validation different for the The main problem here is the overall size of the original search request. Independently of the number of terms in the request, the size in bytes for search requests should be limited. It's already possible to limit the size of the body for http requests but it's hard to set a value that is suitable for bulk and search requests. Maybe we should have a dedicated setting to limit the size of http search requests ? The value could be set aggressively in order to avoid this situation where an abusive query takes a large amount of the resources available in the cluster. |
@jimczi thanks for the response!
|
I am on the fence because we would still need to read and parse the request fully before rejecting it. It's also only working for
Yep that's only one part of the problem. Sending and re-parsing these shard requests is equally costly so I don't think that this added complexity would be helpful. |
This has been open for quite a while, and hasn't had a lot of interest. For now I'm going to close this as something we aren't planning on fixing. We can re-open it later if needed. |
Elasticsearch version (
bin/elasticsearch --version
): 7.10.1Plugins installed: Default
JVM version (
java -version
): 15.0.1OS version (
uname -a
if on a Unix-like system): Linux 7da1c67850ed 4.19.121-linuxkitDescription of the problem including expected versus actual behavior:
Copying over from discussion post
For requests with a very high terms count, Elasticsearch is taking multiple seconds to return an HTTP 400 error. We had an issue with a production system that produced a query that had 250K items in a terms query and this took ~16 seconds to return the HTTP 400. I would expect this validation error to be handled quickly.
Trying to replicate locally, I found that a single instance handled this type of request very quickly. But when running a three node docker cluster, the time it took Elasticsearch to handle this request increased by orders of magnitude as the shard count increased.
The query is just a terms filter against an empty index in the form of:
I documented results in this repo: https://github.com/nickcanz/es-terms-benchmarking but replicating those tables here:
A single docker instance handling a query with 200k terms shows performance to be even as shard count increases.
SHARD_COUNT=1 bash ./make_query.sh
SHARD_COUNT=5 bash ./make_query.sh
SHARD_COUNT=10 bash ./make_query.sh
SHARD_COUNT=25 bash ./make_query.sh
SHARD_COUNT=50 bash ./make_query.sh
SHARD_COUNT=100 bash ./make_query.sh
SHARD_COUNT=150 bash ./make_query.sh
SHARD_COUNT=200 bash ./make_query.sh
On a three node cluster, we see vastly different behavior. Latency increases greatly as shard count increases, so much so that a 200 shard index takes 34 times longer to handle the request. Again, this is against an empty index and just returns an HTTP 400 validation error since it is above the max terms limit, but it still takes ~13 seconds to handle the request.
SHARD_COUNT=1 bash ./make_query.sh
SHARD_COUNT=5 bash ./make_query.sh
SHARD_COUNT=10 bash ./make_query.sh
SHARD_COUNT=25 bash ./make_query.sh
SHARD_COUNT=50 bash ./make_query.sh
SHARD_COUNT=100 bash ./make_query.sh
SHARD_COUNT=150 bash ./make_query.sh
SHARD_COUNT=200 bash ./make_query.sh
Steps to reproduce:
Provide logs (if relevant):
@DaveCTurner was kind enough to add the following to the discuss post
Stack trace
The text was updated successfully, but these errors were encountered: