index.gc_deletes=0s does not behave as expected

[snellm]

If a document is deleted, indexing a new document with the same id within approx 60 seconds of the delete fails with a VersionConflictEngineException.

The scenario where it's necessary to delete and immediately reindex a document is when external versioning is in place and we need to force indexing of an older version of a document - there doesn't appear to be any other way to do this besides deleting the "newer" version before indexing the "older".

[snellm]

We discovered the "index.gc_deletes" setting from trawling the forums (this should be documented on the Delete API page?). Although setting this to 0s helps, it appears VersionConflictEngineException still occur for a short period after the delete.

Perhaps it would be better for delete to honour the gc_deletes setting only if passed a version number to delete? Or is there a better way to deal with the version conflict scenario mentioned above?

[kimchy]

Can you gist a recreation, I know when it might happen, just want something to work with. This is, btw, by design, but it might make sense to allow for your case as well...

[snellm]

Will do. Looking at our case, what we think we'd like (from highest to lowest preference) is:

• A flag on IndexRequest to force indexing regardless of version conflicts
• Make delete with version behave differently from delete without version:
  • delete without a version being passed should not remember the version (ie allow immediate indexing with a different version)
  • delete with version should remember the version for the time period defined by "index.gc_deletes" (ie current behaviour)
• Make it so that if "index.gc_deletes" is set to zero, the version is not remembered at all (currently appears to still be remembered for a small time period)

[snellm]

Here is a gist that recreates the issue: https://gist.github.com/3350884

Output for me is as below - if I uncomment the one second sleep, the VersionConflictEngineException does not occur.

Create a local node with index.gc_deletes set to zero seconds
Indexing a document with external versioning and a version of 2
Deleting the document
Indexing a document with external versioning and a version of 1
Exception in thread "main" org.elasticsearch.index.engine.VersionConflictEngineException: [foo][1] [bar][12345]: version conflict, current [3], provided [1]
    at org.elasticsearch.index.engine.robin.RobinEngine.innerIndex(RobinEngine.java:541)
    at org.elasticsearch.index.engine.robin.RobinEngine.index(RobinEngine.java:486)
    at org.elasticsearch.index.shard.service.InternalIndexShard.index(InternalIndexShard.java:323)
    at org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary(TransportIndexAction.java:206)
    at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:532)
    at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:430)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)

[randunel]

Any updates on this issue? I seem to be experiencing similar issues with version 1.0 beta1

[clintongormley]

The new force version type allows you to override versions in Elasticsearch. http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/docs-index_.html#_version_types