Core: only one optimize operation should run at once

[mikemccand]

Even though we have ThreadPool.OPTIMIZE pool with size=1, if the incoming optimize request does not wait_for_completion, then Lucene's IndexWriter runs the optimize in the background and the request returns immediately, freeing the thread pool to run another optimize.

So if the application submits 10 optimize requests (without wait_for_completion), all 10 will run concurrently, which is bad.

If the optimize is for upgrade, or flush is requested, InternalEngine.optimize does submit a waitForMerges call back to the OPTIMIZE pool, but that's at the end, and so all 10 incoming requests will still run at once I think?

[rjernst]

If the optimize is for upgrade, or flush is requested, InternalEngine.optimize does submit a waitForMerges call back to the OPTIMIZE pool, but that's at the end, and so all 10 incoming requests will still run at once I think?

I think that is correct, or rather whatever other optimize requests have already been queued will be run before the blocking wait for merges for the first request that ran. So I think we need to somehow push the waiting thread to the front of the queue, and always have that regardless of the optimize settings (except for wait_for_completion, which of course just runs in the foreground and holds the single optimize thread).

[rjernst]

Ok, here is my proposal after speaking with Shay:

• For 1.4.3: Change the default for wait_for_completion to true
• For 1.5.0: Remove wait_for_completion (and wait_for_merges in the optimize api)
• For 2.0: Once we have the task api, try to add back some of this async functionality as a long running task that can be managed.

[mikemccand]

+1 for this plan.

Separately, it would be nice if we could simply call IW.forceMerge(), which waits itself. This would fix #8923 ... must we really hold the readLock when calling forceMerge? Anyway, that can be done separately...