Version conflict on document update after elasticsearch update to 7.6.2

[henkepa]

Elasticsearch version (bin/elasticsearch --version): 7.6.2

Plugins installed: [x-pack]

JVM version (java -version): #openjdk version "1.8.0_232"
OpenJDK Runtime Environment (build 1.8.0_232-b09)
OpenJDK 64-Bit Server VM (build 25.232-b09, mixed mode)

OS version (uname -a if on a Unix-like system): Docker running on Redhat 7

Description of the problem including expected versus actual behavior:

Since updating from version 7.4.0 to 7.6.2 I see multiple exceptions on indexing through a pipeline when I set _id and _index in the pipeline. The creation of the documents succeeds but any update of the document fails:

{
        "type" : "version_conflict_engine_exception",
        "reason" : "[1]: version conflict, document already exists (current version [1])",
        "index_uuid" : "va8IOzD0ReqS9Wu0uHLb8g",
        "shard" : "0",
        "index" : "test_ingest_error"
 }

Steps to reproduce:

1. Create a pipeline

PUT _ingest/pipeline/test_ingest_error
{
  "description" : "N/A",
    "processors" : [
    {
        "set" : {
          "field" : "_id",
          "value" : "1"
      }
      },    {
        "set" : {
          "field" : "_index",
          "value" : "test_ingest_error"
        }
      }
    ]
}

{
  "acknowledged" : true
}

2. Create a document via pipeline

POST test_ingest_error/_doc/?pipeline=test_ingest_error
{  
    "prop": "xyz"
}

{
  "_index" : "test_ingest_error",
  "_type" : "_doc",
  "_id" : "1",
  "_version" : 1,
  "result" : "created",
  "_shards" : {
    "total" : 2,
    "successful" : 2,
    "failed" : 0
  },
  "_seq_no" : 0,
  "_primary_term" : 1
}

3. Update the document via pipeline

POST test_ingest_error/_doc/?pipeline=test_ingest_error
 {  
    "prop": "xyzz"

  }

{
  "error" : {
    "root_cause" : [
      {
        "type" : "version_conflict_engine_exception",
        "reason" : "[1]: version conflict, document already exists (current version [1])",
        "index_uuid" : "va8IOzD0ReqS9Wu0uHLb8g",
        "shard" : "0",
        "index" : "test_ingest_error"
      }
    ],
    "type" : "version_conflict_engine_exception",
    "reason" : "[1]: version conflict, document already exists (current version [1])",
    "index_uuid" : "va8IOzD0ReqS9Wu0uHLb8g",
    "shard" : "0",
    "index" : "test_ingest_error"
  },
  "status" : 409
}

[henkepa]

I got the feeback from the support team that the update works with passing op_type=index

POST test_ingest_error/_doc/?pipeline=test_ingest_error&op_type=index
{
"prop": "xyzz"
}

However, my problem remains since I can not immediately update all clients to pass op_type=index.

[elasticmachine]

Pinging @elastic/es-core-features (:Core/Features/Ingest)

[dakrone]

Going to close this since it's so old and very stale. If I am closing it by mistake please comment and we can re-open it.