The entire bulk request fails if any document parsing fails with pipeline configured

[samxbr]

Elasticsearch Version

8.18, 9.0 (potentially other versions too, need to double check)

Problem Description

Normally in a bulk request, if indexing for individual document fails (eg. due to invalid json), the rest of the documents in the bulk request will still succeed, and respond a 200 status, with error details for individual doc in the body. However, if the failing document has a pipeline configured, the entire request will fail with 400 status and none of the documents will be indexed.

This is reported by https://github.com/elastic/sdh-elasticsearch/issues/9486, and this comment pointed we may need to tighten up the error handling in IngestService.

Steps to Reproduce

Create a dummy pipeline

PUT /_ingest/pipeline/test-ds-pl-1
{
  "processors": [
    {
      "append": {
        "field": "name",
        "value": "value"
      }
    }
  ]
}

Bulk index invalid json doc entry without pipeline

PUT /_bulk
{"create":{"_index":"test-ds-1"}}
{"@timestamp":"2024-06-01T00:00:00Z"}
{"create":{"_index":"test-ds-1"}}
{"@timestamp":"2024-06-01T00:00:00Z","@timestamp":"2024-07-01T00:00:00Z"}

• Responds 200, the first doc is indexed, the second doc is not. This is expected.

Bulk index invalid json doc entry with a pipeline

PUT /_bulk?pipeline=test-ds-pl-1
{"create":{"_index":"test-ds-1"}}
{"@timestamp":"2024-06-01T00:00:00Z"}
{"create":{"_index":"test-ds-1"}}
{"@timestamp":"2024-06-01T00:00:00Z","@timestamp":"2024-07-01T00:00:00Z"}

• Responds 400 and no docs are indexed.

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)