Skip to content

Add support to skip retries for certain error codes #1225

@robbavey

Description

@robbavey

In most cases, the Elasticsearch output will attempt to retry sending events that have been rejected in a bulk request to Elasticsearch.

There are certain exceptions to this:

  • Documents that generate a 409 status codes from the bulk response are considered "non_retryable" and are dropped, and a warning message is emitted
  • Documents that generate a status code defined by dlq_custom_codes (400 and 404 by default, with others settable in configuration) are routed to the Dead Letter Queue, where they can be processed later.

Other documents will be retried indefinitely, potentially causing blockages in Logstash systems where the DLQ is not an option.

The Logstash Elasticsearch Output should provide options to handle other cases in a more flexible way.

Example cases:

  • Delays in ingestion may cause documents to be routed to indices that have been transitioned to 'frozen' through ILM policies. These documents will fail to be ingested, returning a 403 status code.
  • While it is possible to add 403 to the dlq_custom_codes configuration property, users may not want to setup a DLQ for the sole purpose of throwing away old documents.

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions