Optimize indexing for the autogenerated ID append-only case

[s1monw]

If elasticsearch controls the ID values as well as the documents
version we can optimize the code that adds / appends the documents
to the index. Essentially we an skip the version lookup for all
documents unless the same document is delivered more than once.

On the lucene level we can simply call IndexWriter#addDocument instead
of #updateDocument but on the Engine level we need to ensure that we deoptimize
the case once we see the same document more than once.

This is done as follows:

1. Mark every request with a timestamp. This is done once on the first node that
   receives a request and is fixed for this request. This can be even the
   machine local time (see why later). The important part is that retry
   requests will have the same value as the original one.
2. In the engine we make sure we keep the highest seen time stamp of "retry" requests.
   This is updated while the retry request has its doc id lock. Call this maxUnsafeAutoIdTimestamp
3. When the engine runs an "optimized" request comes, it compares it's timestamp with the
   current maxUnsafeAutoIdTimestamp (but doesn't update it). If the the request
   timestamp is higher it is safe to execute it as optimized (no retry request with the same
   timestamp has been run before). If not we fall back to "non-optimzed" mode and run the request as a retry one
   and update the maxUnsafeAutoIdTimestamp unless it's been updated already to a higher value

Relates to #19813

[bleskes]

we need to recapture deOptimizeTimestamp in the loop no?

[mikemccand]

I tested performance on indexing geonames (8.6 M docs) on a fast 8 core box on a fast SSD with plenty of RAM (64 GB, vs 4 GB for JVM heap):

before:
 Total docs/sec: 44526.0
 Total docs/sec: 45460.2

after:
 Total docs/sec: 53650.4
 Total docs/sec: 53639.6

~20% speedup, nice :)

[tlrx]

public boolean canHaveDuplicates() ?

[bleskes]

can we explicitly set can have duplicates to false here (and default it true)? or at least assert it is false? (I prefer the first)

[s1monw]

I dont' think we should do this to be honest I added assertions along the way to ensure it's only true if autoGeneratedID is true too also it's documented. We can switch so something like:

enum IDSource {
  External, AutoGenerated, RetryAutoGenerated
}
`` 
to have the different states clear, WDYT?

[s1monw]

@bleskes thx so much for the in-depth reviews, this would have been impossible without that! I will wait for CI to run and push tomorrow morning

[mikemccand]

I re-tested performance gain on latest PR, still ~20% speedup! Nice:

before:

  Total docs/sec: 44299.1
  Total docs/sec: 43192.2  

after:

  Total docs/sec: 53321.2
  Total docs/sec: 52980.7

[bleskes]

@mikemccand w00t. I presume this was without replicas?

[mikemccand]

@bleskes yes: single node with defaults, except 4 GB heap.