New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Graylog doesn't handle Elasticsearch "413 Request Entity Too Large" reply #5091
Comments
mpfz0r
added a commit
that referenced
this issue
Jan 2, 2020
If we try to bulk index a batch of messages that exceeds the elastic search `bulk_max_body_size` setting. (default 100MB) Elastic will respond with an HTTP 413 Entity Too Large error. In this case we retry the request by splitting the message batch in half. When responding with an HTTP 413 error, the server is allowed to close the connection immediatly. This means that our HTTP client (Jest) will simply report with an IOException (Broken pipe) instead of the actual error. This can be avoided by sending the request with an Expect-Continue header, which also avoids sending data that will be discarded later on. Fixes #5091
mpfz0r
added a commit
that referenced
this issue
Jan 2, 2020
If we try to bulk index a batch of messages that exceeds the elastic search `bulk_max_body_size` setting. (default 100MB) Elastic will respond with an HTTP 413 Entity Too Large error. In this case we retry the request by splitting the message batch in half. When responding with an HTTP 413 error, the server is allowed to close the connection immediately. This means that our HTTP client (Jest) will simply report an IOException (Broken pipe) instead of the actual error. This can be avoided by sending the request with an Expect-Continue header, which also avoids sending data that will be discarded later on. Fixes #5091
mpfz0r
added a commit
that referenced
this issue
Jan 3, 2020
If we try to bulk index a batch of messages that exceeds the elastic search `bulk_max_body_size` setting. (default 100MB) Elastic will respond with an HTTP 413 Entity Too Large error. In this case we retry the request by splitting the message batch in half. When responding with an HTTP 413 error, the server is allowed to close the connection immediately. This means that our HTTP client (Jest) will simply report an IOException (Broken pipe) instead of the actual error. This can be avoided by sending the request with an Expect-Continue header, which also avoids sending data that will be discarded later on. Fixes #5091
mpfz0r
added a commit
that referenced
this issue
Jan 10, 2020
If we try to bulk index a batch of messages that exceeds the elastic search `http.max_content_length` setting. (default 100MB) Elastic will respond with an HTTP 413 Entity Too Large error. In this case we retry the request by splitting the message batch in half. When responding with an HTTP 413 error, the server is allowed to close the connection immediately. This means that our HTTP client (Jest) will simply report an IOException (Broken pipe) instead of the actual error. This can be avoided by sending the request with an Expect-Continue header, which also avoids sending data that will be discarded later on. Fixes #5091
bernd
pushed a commit
that referenced
this issue
Jan 10, 2020
If we try to bulk index a batch of messages that exceeds the elastic search `http.max_content_length` setting. (default 100MB) Elastic will respond with an HTTP 413 Entity Too Large error. In this case we retry the request by splitting the message batch in half. When responding with an HTTP 413 error, the server is allowed to close the connection immediately. This means that our HTTP client (Jest) will simply report an IOException (Broken pipe) instead of the actual error. This can be avoided by sending the request with an Expect-Continue header, which also avoids sending data that will be discarded later on. Fixes #5091 * Move JestClient execution with RequestConfig into JestUtils * Please forbiddenapi checker * Correctly handle batches with unevenly sized messages If we have a batch where only the messages at the end will exceed the Entity Too Large limit, we could end up duplicating messages. Thus keep track of the already indexed offset and report it within the EntityTooLargeException. * Make use of Expect: 100-continue header configurable
Thank you for the report. This has been fixed for the upcoming 3.2 version and will also be backported into 3.1. |
bernd
pushed a commit
that referenced
this issue
Jan 10, 2020
If we try to bulk index a batch of messages that exceeds the elastic search `http.max_content_length` setting. (default 100MB) Elastic will respond with an HTTP 413 Entity Too Large error. In this case we retry the request by splitting the message batch in half. When responding with an HTTP 413 error, the server is allowed to close the connection immediately. This means that our HTTP client (Jest) will simply report an IOException (Broken pipe) instead of the actual error. This can be avoided by sending the request with an Expect-Continue header, which also avoids sending data that will be discarded later on. Fixes #5091 * Move JestClient execution with RequestConfig into JestUtils * Please forbiddenapi checker * Correctly handle batches with unevenly sized messages If we have a batch where only the messages at the end will exceed the Entity Too Large limit, we could end up duplicating messages. Thus keep track of the already indexed offset and report it within the EntityTooLargeException. * Make use of Expect: 100-continue header configurable (cherry picked from commit 085930a)
mpfz0r
added a commit
that referenced
this issue
Jan 13, 2020
…#7148) * Handle Request Entity Too Large errors in ElasticSearchOutput (#7071) If we try to bulk index a batch of messages that exceeds the elastic search `http.max_content_length` setting. (default 100MB) Elastic will respond with an HTTP 413 Entity Too Large error. In this case we retry the request by splitting the message batch in half. When responding with an HTTP 413 error, the server is allowed to close the connection immediately. This means that our HTTP client (Jest) will simply report an IOException (Broken pipe) instead of the actual error. This can be avoided by sending the request with an Expect-Continue header, which also avoids sending data that will be discarded later on. Fixes #5091 * Move JestClient execution with RequestConfig into JestUtils * Please forbiddenapi checker * Correctly handle batches with unevenly sized messages If we have a batch where only the messages at the end will exceed the Entity Too Large limit, we could end up duplicating messages. Thus keep track of the already indexed offset and report it within the EntityTooLargeException. * Make use of Expect: 100-continue header configurable (cherry picked from commit 085930a) * Adopt MessagesIT to old IT framwork Also change test to run with a specific index * Skip memory intensive MessagesIT tests on Travis * use getenv not getProperty Co-authored-by: Marco Pfatschbacher <marco@graylog.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
One graylog node lost the connection with elastic cluster.
It seems sometimes, somehow the graylog make bigger batch (based on output_batch_size), then the elasticsearch can allow (ES bulk_max_body_size default 100mb). The graylog can't recognize the received error from ES.
After a graylog service restart can send the logs out to elasticsearch.
After time the GL lost the connection with all ES nodes, and can't store logs. Journal start to grow.
Issue details: https://community.graylog.org/t/elastic-cluster-connection-lost/6189/8
Expected Behavior
GL create smaller size of batch then the ES limit, or handle the error, and refregment the batch if iit receive "413 Request Entity Too Large" error
Current Behavior
GL makes a batch, try to send to ES, ES denies it, ES send TCP reset, GL try another ES with the same batch.
GL: log:
slice of TCPdump
// it contains the GL uncommited log entries, sample every 5min

Possible Solution
Recreate batch if ES denies with error 413 Request Entity Too Large
Chech the bulk_max_body_size before communication and don't create bigger batch then the bulk_max_body_size
Or make a new variable in server.conf to set the max size of batch size (in KB or MB)
Steps to Reproduce (for bugs)
log_numberlog_size > ES bulk_max_body_size && log_number < GL output_batch_size
eg. 1000 piece1M log
Context
Your Environment
The text was updated successfully, but these errors were encountered: