Use FileChannel in FSDirectory [LUCENE-2871]

[asfimport]

Explore using FileChannel in FSDirectory to see if it improves write operations performance

---

Migrated from LUCENE-2871 by Shay Banon (@kimchy), updated Jan 20 2011
Attachments: LUCENE-2871.patch (versions: 2)

[asfimport]

Shay Banon (@kimchy) (migrated from JIRA)

Patch supporting using file channel to write. FSDirectory still retain the ability to use RAF for writes.

FSDirectory#setUseChannelOutput: Allows to revert back to use RAF by setting to false.
FSDirectory#setCacheChannelBuffers: Allow to control if, when using file channel, buffers should be cached.

[asfimport]

Michael McCandless (@mikemccand) (migrated from JIRA)

I attempted to benchmark this patch, testing indexing time for Wikipedia docs, but I hit this exception:

Exception in thread "Lucene Merge Thread #49" org.apache.lucene.index.MergePolicy$MergeException: java.nio.BufferOverflowException
	at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:507)
	at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:472)
Caused by: java.nio.BufferOverflowException
	at java.nio.Buffer.nextPutIndex(Buffer.java:501)
	at java.nio.DirectByteBuffer.putLong(DirectByteBuffer.java:745)
	at org.apache.lucene.store.FSDirectory$FSChannelIndexOutput.writeLong(FSDirectory.java:706)
	at org.apache.lucene.index.codecs.PrefixCodedTermsWriter.writeTrailer(PrefixCodedTermsWriter.java:146)
	at org.apache.lucene.index.codecs.PrefixCodedTermsWriter.close(PrefixCodedTermsWriter.java:129)
	at org.apache.lucene.index.PerFieldCodecWrapper$FieldsWriter.close(PerFieldCodecWrapper.java:80)
	at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:559)
	at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:120)
	at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3084)
	at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:2768)
	at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:379)
	at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:447)

This was after adding ~6 M docs... the 100K doc index built just fine. Shay any ideas?

[asfimport]

Shay Banon (@kimchy) (migrated from JIRA)

Strange, did not get it when running the tests, will try and find out why it can happen.

[asfimport]

Michael McCandless (@mikemccand) (migrated from JIRA)

Yeah, me neither – tests all pass when I force dir to eg NIOFSDir, and, my benchmark runs on the 100K index; just fails for the 10M index... curious.

[asfimport]

Uwe Schindler (@uschindler) (migrated from JIRA)

Looking at the current patch, the class seems wrong. In my opinion, this should be only in NIOFSDirectory. SimpleFSDir should only use RAF.

[asfimport]

Shay Banon (@kimchy) (migrated from JIRA)

Fixed Buffer Overflow exception (I hope, can't really recreate it, mike can...). Also, per the IRC discussion, made SimpleFSDirectory default to not use the file channel output, while NIO and MMap default to use it. One can still control if it will be used or not using the setter method.

[asfimport]

Shay Banon (@kimchy) (migrated from JIRA)

Looking at the current patch, the class seems wrong. In my opinion, this should be only in NIOFSDirectory. SimpleFSDir should only use RAF.

Its a good question, not sure what to do with it. Here is the problem. The channel output can be used with all 3 FS dirs (simple, nio, and mmap), and actually might make sense to be used even with SimpleFS (i.e. using non nio to read, but use file channel to write). In order to have all of them supported, currently, the simplest way is to put it in the base class so code will be shared. On IRC, a discussion was made to externalize the outputs and inputs, and then one can more easily pick and choose, but I think this will belong on a different patch.

[asfimport]

Earwin Burrfoot (migrated from JIRA)

Before arguing where to put this new IndexOutput, I think it's wise to have a benchmark proving we need it at all.
I have serious doubts FileChannel's going to outperform RAF.write(). Why should it?
And for the purporses of benchmark it can be anywhere.

[asfimport]

Shay Banon (@kimchy) (migrated from JIRA)

Agreed Earwin, lets first see if it make sense, this is just an experiment and might not make sense for single threaded writes.

[asfimport]

Michael McCandless (@mikemccand) (migrated from JIRA)

OK – I was able to index 10M docs w/ the new patch. And search results are identical.

But the indexing time on trunk vs the patch were nearly identical – 536.80 sec (trunk) and 536.06 (w/ patch). But, this is on a fast machine, lots of RAM (so writes go straight to buffer cache) and an SSD, using 6 indexing threads.