RestStatus of "too_many_buckets_exception"

[delvedor]

Hello!
I'm running the integration test for the JavaScript client with Elasticsearch version 7.
The test highlighted here is failing because the status code of the error is 503, and every client will mark the connection as dead and retry with a new one if the status code is 502, 503 or 504.

Following the error that I'm getting back from ES:

{  
  "body":{  
    "error":{  
      "root_cause":[  
        {  
          "type":"too_many_buckets_exception",
          "reason":"Trying to create too many buckets. Must be less than or equal to: [3] but was [8]. This limit can be set by changing the [search.max_buckets] cluster level setting.",
          "max_buckets":3
        }
      ],
      "type":"search_phase_execution_exception",
      "reason":"all shards failed",
      "phase":"query",
      "grouped":true,
      "failed_shards":[  
        {  
          "shard":0,
          "index":"test",
          "node":"90I7wSEvQwOuQvfE6GYpJg",
          "reason":{  
            "type":"too_many_buckets_exception",
            "reason":"Trying to create too many buckets. Must be less than or equal to: [3] but was [8]. This limit can be set by changing the [search.max_buckets] cluster level setting.",
            "max_buckets":3
          }
        }
      ]
    },
    "status":503
  },
  "statusCode":503,
  "headers":{  
    "content-type":"application/json; charset=UTF-8",
    "content-length":"687"
  },
  "warnings":null
}

This might be related to #31986.
cc @elastic/es-clients

[elasticmachine]

Pinging @elastic/es-analytics-geo

[jimczi]

I agree that a 503 might not be appropriate here. search.max_buckets is a cluster setting so the query will always fail unless the value of the setting is changed by an admin. I wonder if the bad request (400) error code is appropriate in this case since retrying the query should not be an option ? The other option is to return an error 500 but I am not sure it conveys the message clearly.

[polyfractal]

We discussed this and decided to follow how CircuitBreaker was modified and throw a 429 instead of 503.

We'd also like to add a "durability field" similar to what CB did (something like CB's "durability": "PERMANENT"), to indicate the exception is non-retryable since it's the query structure that tripped the max_buckets, rather than just a transient failure.

[polyfractal]

Just skimming through status codes, 422 Unprocessable Entity may be applicable too:

The HyperText Transfer Protocol (HTTP) 422 Unprocessable Entity response status code indicates that the server understands the content type of the request entity, and the syntax of the request entity is correct, but it was unable to process the contained instructions.

Important: The client should not repeat this request without modification.

[mhugo]

Hi, I encountered the same problem with the Python client (on 6.8 and 7.9) that retries many times when a too_many_buckets_exception occurs because of the 503 return status. 4xx seems more appropriate indeed. Is there any progress on this topic ?

[elasticsearchmachine]

Pinging @elastic/es-analytics-geo (Team:Analytics)