Skip to content

Optimistic Concurrency Control Support #271

@HDDTHR

Description

@HDDTHR

Is your feature request related to a problem? Please describe.
Although the Bulk API supports the if_seq_no and if_primary_term options for each document in the request below, the OpenSearch output plugin doesn't provide an option to set those values for the request.

POST _bulk
{"index":{"_index":"some-index","_id":"1", "if_seq_no": 100 "if_primary_term": 20 }}
{"amount": 500 }

Describe the solution you'd like
Adding two new options to support Optimistic Concurrency Control for the output plugin. It could look like this.

        opensearch {
            id => "os-output"
            hosts => "..."
            user => "..."
            password => "..."
            index => "some-index"
            document_id => "..."
            sequence_number => "[@metadata][_seq_no]"
            primary_term => "[@metadata][_seq_no]"
        }

Describe alternatives you've considered
I had to figure out a workaround, where I fetch the _version field alongside the document from OpenSearch, manually increment it by 1, and configure my output plugin as shown below.

        opensearch {
            id => "os-output"
            hosts => "..."
            user => "..."
            password => "..."
            index => "some-index"
            document_id => "..."
            version => "%{[@metadata][_version]}"
            version_type => "external"
        }

While this does work, I would rather not have to manage my version externally (and introduce unnecessary points of failure), as the only logic I need is to update the document in OpenSearch if there hasn't been any updates while the pipeline is running.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementIncreases software capabilities beyond original client specifications

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions