Fix TestIndexWriterWithThreads exceeding max file handles.

[dweiss]

TestIndexWriterWithThreads.testCloseWithThreads occasionally throws "Too many open files" from HandleLimitFS. This test is designed to run multiple threads in parallel, each thread adding documents and performing merges, while the main thread closes the index writer forcefully.

I rewrote the test slightly but didn't want to add too much thread coordination. I also bumped the HandleLimitFS to 4x the default limit. The test passes for me with -Ptests.iters=100 with no problems after that.

[uschindler]

Instead of raising the file limit, can we not reduce the number of threads?

Uwe

[dweiss]

This won't work. The problem isn't the number of threads but that they add documents, uncoordinated. Even a single thread could - in theory - add so many documents and create so many intermediate segments that it'll overflow the limit. The test itself is designed to work this way - to allow indexing threads run uncontrollably until the index writer is closed... I don't think there is any reasonable way to throttle the speed with which those background threads index new documents without negating the very reason this test exists.

[uschindler]

Ok,
I was just afraid that it fails on a real filesystem. Can we maybe change this test so it never uses real filesystem?
So hardcode to Bytebuffersdirectory.

[dweiss]

I did think about this too but again - wasn't this meant to be checking all the layers, including directory implementations? This seems to be designed to check "what happens if I close the index writer underneath running indexing threads".