-
Notifications
You must be signed in to change notification settings - Fork 307
Closed
Description
In most cases, the Elasticsearch output will attempt to retry sending events that have been rejected in a bulk request to Elasticsearch.
There are certain exceptions to this:
- Documents that generate a
409
status codes from the bulk response are considered "non_retryable" and are dropped, and a warning message is emitted - Documents that generate a status code defined by dlq_custom_codes (
400
and404
by default, with others settable in configuration) are routed to the Dead Letter Queue, where they can be processed later.
Other documents will be retried indefinitely, potentially causing blockages in Logstash systems where the DLQ is not an option.
The Logstash Elasticsearch Output should provide options to handle other cases in a more flexible way.
Example cases:
- Delays in ingestion may cause documents to be routed to indices that have been transitioned to 'frozen' through ILM policies. These documents will fail to be ingested, returning a
403
status code. - While it is possible to add
403
to thedlq_custom_codes
configuration property, users may not want to setup a DLQ for the sole purpose of throwing away old documents.