Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Update with retry_on_conflict stays stuck in an infinite loop when sent as a part of bulk #11152

Open
raghuvanshraj opened this issue Nov 10, 2023 · 0 comments
Assignees
Labels
bug Something isn't working Indexing Indexing, Bulk Indexing and anything related to indexing

Comments

@raghuvanshraj
Copy link
Contributor

Describe the bug
When an update is sent as a part of bulk, the request stays stuck in an infinite loop in TransportShardBulkAction due to repeated retries. This code pointer is expected to limit the number of retries to retry_on_conflict specified by the user in the bulk request, but the retryCounter in the BulkPrimaryExecutionContext is never incremented in the resetForExecutionForRetry method. In a scenario where there are repeated conflicts for an update, the loop in TransportShardBulkAction remains stuck forever. Note that this same behaviour is not seen when the _update API is invoked and a single document is updated because for the _update API, retries are handled in TransportUpdateAction.

To Reproduce
We need to throw VersionConflictEngineExceptions repeatedly for this to show up. I have created a remote branch on my fork where I have modified the update code to always throw VersionConflictEngineExceptions: https://github.com/raghuvanshraj/OpenSearch/tree/retry-on-conflict-testing

Steps to reproduce the behavior:

  1. Clone the branch linked above
  2. Bring up the opensearch process
  3. Create an index
  4. Ingest a single document on the index
  5. Update the document with the bulk API with retry_on_conflict set in the update request. Sample:
{ "update" : { "_index" : "{{index_name}}", "retry_on_conflict": 3, "_id": "IYGCtIsBspX0Krzt2kus" } }
{ "doc": { "counter": 3 } }

Expected behavior
The expected behavior in this case would be for retry_on_conflict to be honored and for the request to be succeeded/failed gracefully.

Plugins
NA

Screenshots
NA

Host/Environment (please complete the following information):
NA

Additional context
NA

@raghuvanshraj raghuvanshraj added bug Something isn't working untriaged labels Nov 10, 2023
@raghuvanshraj raghuvanshraj self-assigned this Nov 10, 2023
@sarthakaggarwal97 sarthakaggarwal97 added the Indexing Indexing, Bulk Indexing and anything related to indexing label Nov 10, 2023
@raghuvanshraj raghuvanshraj changed the title [BUG] Update with retry_on_conflict stays stuck in an infinite loop when sent as a part of bulk [BUG] Update with retry_on_conflict stays stuck in an infinite loop when sent as a part of bulk Nov 10, 2023
@raghuvanshraj raghuvanshraj changed the title [BUG] Update with retry_on_conflict stays stuck in an infinite loop when sent as a part of bulk [BUG] Update with retry_on_conflict stays stuck in an infinite loop when sent as a part of bulk Nov 10, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working Indexing Indexing, Bulk Indexing and anything related to indexing
Projects
None yet
Development

No branches or pull requests

2 participants