Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

LUCENE-9996: Reduce RAM usage of DWPT for a single document. #184

Merged
merged 3 commits into from Jun 18, 2021

Conversation

jpountz
Copy link
Contributor

@jpountz jpountz commented Jun 15, 2021

With this change, doc-value terms dictionaries use a shared ByteBlockPool
across all fields, and points, binary doc values and doc-value ordinals use
slightly smaller page sizes.

With this change, doc-value terms dictionaries use a shared `ByteBlockPool`
across all fields, and points, binary doc values and doc-value ordinals use
slightly smaller page sizes.
@jpountz
Copy link
Contributor Author

jpountz commented Jun 15, 2021

On this simple test, memory usage for a single doc in the DWPT goes from 9.6MB to 1.4MB.

import org.apache.lucene.document.Document;
import org.apache.lucene.document.Field.Store;
import org.apache.lucene.document.IntPoint;
import org.apache.lucene.document.SortedNumericDocValuesField;
import org.apache.lucene.document.SortedSetDocValuesField;
import org.apache.lucene.document.StoredField;
import org.apache.lucene.document.StringField;
import org.apache.lucene.index.IndexWriter;
import org.apache.lucene.index.IndexWriterConfig;
import org.apache.lucene.store.ByteBuffersDirectory;
import org.apache.lucene.util.BytesRef;

public class IWBuffer {

  public static void main(String[] args) throws Exception {
    try (ByteBuffersDirectory dir = new ByteBuffersDirectory();
        IndexWriter w = new IndexWriter(dir, new IndexWriterConfig().setRAMBufferSizeMB(1000))) {
      Document doc = new Document();
      for (int i = 0; i < 100; ++i) {
        String keywordFieldName = "keyword" + i;
        doc.add(new StringField(keywordFieldName, "Lucene", Store.YES));
        doc.add(new SortedSetDocValuesField(keywordFieldName, new BytesRef("Lucene")));

        String numericFieldName = "numeric" + i;
        doc.add(new IntPoint(numericFieldName, 4));
        doc.add(new SortedNumericDocValuesField(numericFieldName, 4));
        doc.add(new StoredField(numericFieldName, 4));
      }
      w.addDocument(doc);
      System.out.println(w.ramBytesUsed() / 1024. / 1024.);
    }
  }

}

Copy link
Member

@dnhatn dnhatn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@jpountz jpountz merged commit 1365156 into apache:main Jun 18, 2021
@jpountz jpountz deleted the lucene9996 branch June 18, 2021 07:17
mikemccand pushed a commit to mikemccand/lucene that referenced this pull request Sep 3, 2021
)

With this change, doc-value terms dictionaries use a shared `ByteBlockPool`
across all fields, and points, binary doc values and doc-value ordinals use
slightly smaller page sizes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants