-
Notifications
You must be signed in to change notification settings - Fork 83
Description
Is your feature request related to a problem? Please describe.
Although the Bulk API supports the if_seq_no
and if_primary_term
options for each document in the request below, the OpenSearch output plugin doesn't provide an option to set those values for the request.
POST _bulk
{"index":{"_index":"some-index","_id":"1", "if_seq_no": 100 "if_primary_term": 20 }}
{"amount": 500 }
Describe the solution you'd like
Adding two new options to support Optimistic Concurrency Control for the output plugin. It could look like this.
opensearch {
id => "os-output"
hosts => "..."
user => "..."
password => "..."
index => "some-index"
document_id => "..."
sequence_number => "[@metadata][_seq_no]"
primary_term => "[@metadata][_seq_no]"
}
Describe alternatives you've considered
I had to figure out a workaround, where I fetch the _version
field alongside the document from OpenSearch, manually increment it by 1, and configure my output plugin as shown below.
opensearch {
id => "os-output"
hosts => "..."
user => "..."
password => "..."
index => "some-index"
document_id => "..."
version => "%{[@metadata][_version]}"
version_type => "external"
}
While this does work, I would rather not have to manage my version externally (and introduce unnecessary points of failure), as the only logic I need is to update the document in OpenSearch if there hasn't been any updates while the pipeline is running.