Skip to content

Optimize byte array writes in OutputStreamIndexOutput to remove unnecessary overhead#16119

Open
neoremind wants to merge 2 commits into
apache:mainfrom
neoremind:write_bytes_io
Open

Optimize byte array writes in OutputStreamIndexOutput to remove unnecessary overhead#16119
neoremind wants to merge 2 commits into
apache:mainfrom
neoremind:write_bytes_io

Conversation

@neoremind
Copy link
Copy Markdown
Contributor

@neoremind neoremind commented May 25, 2026 •
edited
Loading

Inspired by #13543 for write(int), I was profiling the write path and noticed a similar optimization can be applied to writeBytes, this can eliminate unnecessary growIfNeeded checks and the large-byte-array bypass checks.

Why this is safe:

  1. XBufferedOutputStream uses a fixed 8KB buffer, it never resizes, growIfNeeded is pure overhead.
  2. The fallback to super.write(b, off, len) retains the existing behavior for large byte arrays that should bypass buffering entirely and chunk them later for real I/O.

Verification

Since Lucene indexing is CPU-intensive and the I/O path isn't the time-consuming part, there is no significant processing time difference in luceneutil nightly benchmarks. But I did profiling with JFR and verified the overhead is completely removed.

Note that I can only sample growIfNeeded since it's a method call, the large-byte-array bypass check is inlined and hard to measure, so the real benefit should be slightly better than what the numbers below show.

Wikipedia medium

33M docs, single-thread, with dv, wait for commit/merge, use CMS and TieredMergePolicy

growIfNeeded samples Total CPU samples Ratio
Baseline 26 409,250 0.01%

See Appendix 2. for detailed call stack captured.

flamegraph link
log link

What to call out is that writeBytes is not the dominanting write op in OutputStreamIndexOutput, only 5% of total calls. But the savings are still positive. In a 33M wikipedia indexing, it removes 245M unnecessary growIfNeeded and large-byte-array bypass checks.

I intercepted OutputStreamIndexOutput to count every write call (see how-to in Appendix 1). The stats show writeInt dominates by call count (61.4%), and writeBytes handles nearly half the total bytes despite being only 5% of calls. And 93% of writeBytes calls are less than 128 bytes, they all hit the optimization.

WriteStats: totalOps=4,875,075,375 totalBytes=26,174,155,590
  writeByte:  1,627,290,133 ( 33.4% calls,   6.2% bytes)
  writeBytes: 245,247,290 (  5.0% calls,  47.9% bytes, avgLen=51)
  writeInt:   2,991,534,308 ( 61.4% calls,  45.7% bytes)
  writeLong:   5,474,411 (  0.1% calls,   0.2% bytes)
  writeShort:  5,529,233 (  0.1% calls,   0.0% bytes)
  writeBytes size distribution:
    [1-16]:    164,778,837
    [17-128]:  65,456,422
    [129-1K]:  14,993,877
    [1K-8K]:   1,722
    [>8K]:     16,432

KNN indexing

1M 1024-dim float32 vectors, wait for commit/merge, use CMS, force merge at last

growIfNeeded samples Total CPU samples Ratio
Baseline 4 1,943,015 0.0002%

See Appendix 2. for detailed call stack captured.

flamegraph link
log link

For knn, writeBytes dominates at 78.4% of all write calls, indexing 1M vectors involves tens of millions of writeBytes ops, and all of them benefit from the fast path. But since knn is super CPU-intensive with vectorized dot product dominating the profiling, the small improvement diminishes as it's barely visible in the samples, but still a micro-level win.

During multi-thread indexing with 1024-dim float32 vector, rerank to improve precision. Wait for commit and merge to complete:

WriteStats: totalOps=55,596,839 totalBytes=12,931,045,866
  writeByte:  11,985,442 ( 21.6% calls,   0.1% bytes)
  writeBytes: 43,597,865 ( 78.4% calls,  99.9% bytes, avgLen=296)
  writeInt:        4,752 (  0.0% calls,   0.0% bytes)
  writeLong:       8,574 (  0.0% calls,   0.0% bytes)
  writeShort:        206 (  0.0% calls,   0.0% bytes)
  writeBytes size distribution:
    [1-16]:    40,481,940
    [17-128]:  7,426
    [129-1K]:  32,581
    [1K-8K]:   3,075,918
    [>8K]:     0

Follow-up merge to 1 segment:

WriteStats: totalOps=21,901,875 totalBytes=4,252,427,320
  writeByte:   4,719,616 ( 21.5% calls,   0.1% bytes)
  writeBytes: 17,180,254 ( 78.4% calls,  99.9% bytes, avgLen=247)
  writeInt:           38 (  0.0% calls,   0.0% bytes)
  writeLong:       1,966 (  0.0% calls,   0.0% bytes)
  writeShort:          1 (  0.0% calls,   0.0% bytes)
  writeBytes size distribution:
    [1-16]:    16,169,470
    [17-128]:  478
    [129-1K]:  9,791
    [1K-8K]:   1,000,515
    [>8K]:     0

Write path deep dive

FSDirectory.FSIndexOutput writes data through a layered stack:

Lucene write call (writeByte, writeBytes, writeShort, writeInt, writeLong)
  → XBufferedOutputStream          (heap byte[] buffer, 8KB)
    → CheckedOutputStream          (CRC32 checksum, inline on flush)
      → FilterOutputStream         (chunks writes to ≤ CHUNK_SIZE)
        → ChannelOutputStream
          → FileChannelImpl.write(HeapByteBuffer)
            → sun.nio.ch.IOUtil.write()
              → getTemporaryDirectBuffer()   (thread-local cached DirectByteBuffer)
                → memcpy heap → direct
              → pwrite() syscall             (kernel page cache → disk)

For the common case that small writes that fit in the buffer, do System.arraycopy, no unnecessary checks. For large arrays that the buffer can't hold, fall to BufferedOutputStream.write() which flushes the buffer and writes directly to the underlying stream, the FilterOutputStream in FSDirectory then chunks those large writes to less than 8K before hitting the file channel, preventing unwanted larger native buffer allocation. This is true, so I've also updated the stale CHUNK_SIZE comment as the original comment is a bit misleading.

wite-path-deep-dive

Appendix 1. Statistical analysis method of lucene write path

Write statistics were captured using a modified luceneutil that intercepts all write calls and performs statistical analysis after indexing completes: luceneutil commit.

Appendix 2. Detailed call stacks hitting growIfNeeded captured by JFR

Baseline top stacks of wikipedia 33MM

--- Count: 3 ---
java.io.BufferedOutputStream.growIfNeeded() line: 138
java.io.BufferedOutputStream.write() line: 186
org.apache.lucene.store.OutputStreamIndexOutput.writeBytes() line: 59
org.apache.lucene.store.ByteBuffersDataOutput.copyTo() line: 338
org.apache.lucene.codecs.lucene104.Lucene104PostingsWriter.flushDocBlock() line: 491
org.apache.lucene.codecs.lucene104.Lucene104PostingsWriter.finishTerm() line: 576
org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm() line: 172
org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter$TermsWriter.write() line: 988
org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter.write() line: 390
org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.write() line: 161
org.apache.lucene.index.FreqProxTermsWriter.flush() line: 133
org.apache.lucene.index.IndexingChain.flush() line: 361
org.apache.lucene.index.DocumentsWriterPerThread.flush() line: 492
org.apache.lucene.index.DocumentsWriter.doFlush() line: 541
org.apache.lucene.index.DocumentsWriter.postUpdate() line: 404
org.apache.lucene.index.DocumentsWriter.updateDocuments() line: 450
org.apache.lucene.index.IndexWriter.updateDocuments() line: 1600
org.apache.lucene.index.IndexWriter.updateDocument() line: 1882
org.apache.lucene.index.IndexWriter.addDocument() line: 1491
perf.IndexThreads$IndexThread.run() line: 419

--- Count: 2 ---
java.io.BufferedOutputStream.growIfNeeded() line: 135
java.io.BufferedOutputStream.write() line: 186
org.apache.lucene.store.OutputStreamIndexOutput.writeBytes() line: 59
org.apache.lucene.store.RateLimitedIndexOutput.writeBytes() line: 60
org.apache.lucene.util.bkd.BKDWriter.writeLeafBlockPackedValuesRange() line: 1496
org.apache.lucene.util.bkd.BKDWriter.writeHighCardinalityLeafBlockPackedValues() line: 1422
org.apache.lucene.util.bkd.BKDWriter.writeLeafBlockPackedValues() line: 1358
org.apache.lucene.util.bkd.BKDWriter$OneDimensionBKDWriter.writeLeafBlock() line: 859
org.apache.lucene.util.bkd.BKDWriter$OneDimensionBKDWriter.add() line: 754
org.apache.lucene.util.bkd.BKDWriter.merge() line: 666
org.apache.lucene.codecs.lucene90.Lucene90PointsWriter.merge() line: 279
org.apache.lucene.index.SegmentMerger.mergePoints() line: 197
org.apache.lucene.index.SegmentMerger$$Lambda.0x000000004f1ca8e0.merge() line: -1
org.apache.lucene.index.SegmentMerger.mergeWithLogging() line: 315
org.apache.lucene.index.SegmentMerger.merge() line: 155
org.apache.lucene.index.IndexWriter.mergeMiddle() line: 5322
org.apache.lucene.index.IndexWriter.merge() line: 4785
org.apache.lucene.index.IndexWriter$IndexWriterMergeSource.merge() line: 6590
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge() line: 661
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run() line: 726

--- Count: 2 ---
java.io.BufferedOutputStream.growIfNeeded() line: 137
java.io.BufferedOutputStream.write() line: 186
org.apache.lucene.store.OutputStreamIndexOutput.writeBytes() line: 59
org.apache.lucene.store.ByteBuffersDataOutput.copyTo() line: 338
org.apache.lucene.codecs.lucene104.Lucene104PostingsWriter.flushDocBlock() line: 491
org.apache.lucene.codecs.lucene104.Lucene104PostingsWriter.finishTerm() line: 576
org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm() line: 172
org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter$TermsWriter.write() line: 988
org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter.write() line: 390
org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.write() line: 161
org.apache.lucene.index.FreqProxTermsWriter.flush() line: 133
org.apache.lucene.index.IndexingChain.flush() line: 361
org.apache.lucene.index.DocumentsWriterPerThread.flush() line: 492
org.apache.lucene.index.DocumentsWriter.doFlush() line: 541
org.apache.lucene.index.DocumentsWriter.postUpdate() line: 404
org.apache.lucene.index.DocumentsWriter.updateDocuments() line: 450
org.apache.lucene.index.IndexWriter.updateDocuments() line: 1600
org.apache.lucene.index.IndexWriter.updateDocument() line: 1882
org.apache.lucene.index.IndexWriter.addDocument() line: 1491
perf.IndexThreads$IndexThread.run() line: 419

--- Count: 2 ---
java.io.BufferedOutputStream.growIfNeeded() line: 135
java.io.BufferedOutputStream.write() line: 186
org.apache.lucene.store.OutputStreamIndexOutput.writeBytes() line: 59
org.apache.lucene.util.bkd.BKDWriter.writeLeafBlockPackedValuesRange() line: 1496
org.apache.lucene.util.bkd.BKDWriter.writeHighCardinalityLeafBlockPackedValues() line: 1422
org.apache.lucene.util.bkd.BKDWriter.writeLeafBlockPackedValues() line: 1358
org.apache.lucene.util.bkd.BKDWriter$OneDimensionBKDWriter.writeLeafBlock() line: 859
org.apache.lucene.util.bkd.BKDWriter$OneDimensionBKDWriter.add() line: 754
org.apache.lucene.util.bkd.BKDWriter$2.visit() line: 609
org.apache.lucene.index.PointValuesWriter$1.visitDocValues() line: 209
org.apache.lucene.util.bkd.BKDWriter.writeField1Dim() line: 604
org.apache.lucene.util.bkd.BKDWriter.writeField() line: 449
org.apache.lucene.codecs.lucene90.Lucene90PointsWriter.writeField() line: 156
org.apache.lucene.index.PointValuesWriter.flush() line: 317
org.apache.lucene.index.IndexingChain.writePoints() line: 408
org.apache.lucene.index.IndexingChain.flush() line: 318
org.apache.lucene.index.DocumentsWriterPerThread.flush() line: 492
org.apache.lucene.index.DocumentsWriter.doFlush() line: 541
org.apache.lucene.index.DocumentsWriter.postUpdate() line: 404
org.apache.lucene.index.DocumentsWriter.updateDocuments() line: 450
org.apache.lucene.index.IndexWriter.updateDocuments() line: 1600
org.apache.lucene.index.IndexWriter.updateDocument() line: 1882
org.apache.lucene.index.IndexWriter.addDocument() line: 1491
perf.IndexThreads$IndexThread.run() line: 419

--- Count: 2 ---
java.io.BufferedOutputStream.growIfNeeded() line: 138
java.io.BufferedOutputStream.write() line: 186
org.apache.lucene.store.OutputStreamIndexOutput.writeBytes() line: 59
org.apache.lucene.util.bkd.BKDWriter.writeLeafBlockPackedValuesRange() line: 1496
org.apache.lucene.util.bkd.BKDWriter.writeHighCardinalityLeafBlockPackedValues() line: 1422
org.apache.lucene.util.bkd.BKDWriter.writeLeafBlockPackedValues() line: 1358
org.apache.lucene.util.bkd.BKDWriter$OneDimensionBKDWriter.writeLeafBlock() line: 859
org.apache.lucene.util.bkd.BKDWriter$OneDimensionBKDWriter.add() line: 754
org.apache.lucene.util.bkd.BKDWriter$2.visit() line: 609
org.apache.lucene.index.PointValuesWriter$1.visitDocValues() line: 209
org.apache.lucene.util.bkd.BKDWriter.writeField1Dim() line: 604
org.apache.lucene.util.bkd.BKDWriter.writeField() line: 449
org.apache.lucene.codecs.lucene90.Lucene90PointsWriter.writeField() line: 156
org.apache.lucene.index.PointValuesWriter.flush() line: 317
org.apache.lucene.index.IndexingChain.writePoints() line: 408
org.apache.lucene.index.IndexingChain.flush() line: 318
org.apache.lucene.index.DocumentsWriterPerThread.flush() line: 492
org.apache.lucene.index.DocumentsWriter.doFlush() line: 541
org.apache.lucene.index.DocumentsWriter.postUpdate() line: 404
org.apache.lucene.index.DocumentsWriter.updateDocuments() line: 450
org.apache.lucene.index.IndexWriter.updateDocuments() line: 1600
org.apache.lucene.index.IndexWriter.updateDocument() line: 1882
org.apache.lucene.index.IndexWriter.addDocument() line: 1491
perf.IndexThreads$IndexThread.run() line: 419

--- Count: 2 ---
java.io.BufferedOutputStream.growIfNeeded() line: 138
java.io.BufferedOutputStream.write() line: 186
org.apache.lucene.store.OutputStreamIndexOutput.writeBytes() line: 59
org.apache.lucene.store.RateLimitedIndexOutput.writeBytes() line: 60
org.apache.lucene.store.DataOutput.writeBytes() line: 54
org.apache.lucene.codecs.lucene104.PForUtil.encode() line: 104
org.apache.lucene.codecs.lucene104.Lucene104PostingsWriter.addPosition() line: 342
org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm() line: 159
org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter$TermsWriter.write() line: 988
org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter.write() line: 390
org.apache.lucene.codecs.FieldsConsumer.merge() line: 94
org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.merge() line: 190
org.apache.lucene.index.SegmentMerger.mergeTerms() line: 223
org.apache.lucene.index.SegmentMerger$$Lambda.0x000000004f1c48b8.merge() line: -1
org.apache.lucene.index.SegmentMerger.mergeWithLogging() line: 315
org.apache.lucene.index.SegmentMerger.merge() line: 147
org.apache.lucene.index.IndexWriter.mergeMiddle() line: 5322
org.apache.lucene.index.IndexWriter.merge() line: 4785
org.apache.lucene.index.IndexWriter$IndexWriterMergeSource.merge() line: 6590
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge() line: 661
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run() line: 726

--- Count: 1 ---
java.io.BufferedOutputStream.growIfNeeded() line: 138
java.io.BufferedOutputStream.write() line: 186
org.apache.lucene.store.OutputStreamIndexOutput.writeBytes() line: 59
org.apache.lucene.store.DataOutput.writeBytes() line: 54
org.apache.lucene.codecs.lucene104.PForUtil.encode() line: 104
org.apache.lucene.codecs.lucene104.Lucene104PostingsWriter.addPosition() line: 342
org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm() line: 159
org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter$TermsWriter.write() line: 988
org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter.write() line: 390
org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.write() line: 161
org.apache.lucene.index.FreqProxTermsWriter.flush() line: 133
org.apache.lucene.index.IndexingChain.flush() line: 361
org.apache.lucene.index.DocumentsWriterPerThread.flush() line: 492
org.apache.lucene.index.DocumentsWriter.doFlush() line: 541
org.apache.lucene.index.DocumentsWriter.postUpdate() line: 404
org.apache.lucene.index.DocumentsWriter.updateDocuments() line: 450
org.apache.lucene.index.IndexWriter.updateDocuments() line: 1600
org.apache.lucene.index.IndexWriter.updateDocument() line: 1882
org.apache.lucene.index.IndexWriter.addDocument() line: 1491
perf.IndexThreads$IndexThread.run() line: 419

--- Count: 1 ---
java.io.BufferedOutputStream.growIfNeeded() line: 142
java.io.BufferedOutputStream.write() line: 186
org.apache.lucene.store.OutputStreamIndexOutput.writeBytes() line: 59
org.apache.lucene.store.ByteBuffersDataOutput.copyTo() line: 338
org.apache.lucene.codecs.lucene104.Lucene104PostingsWriter.flushDocBlock() line: 491
org.apache.lucene.codecs.lucene104.Lucene104PostingsWriter.finishTerm() line: 576
org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm() line: 172
org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter$TermsWriter.write() line: 988
org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter.write() line: 390
org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.write() line: 161
org.apache.lucene.index.FreqProxTermsWriter.flush() line: 133
org.apache.lucene.index.IndexingChain.flush() line: 361
org.apache.lucene.index.DocumentsWriterPerThread.flush() line: 492
org.apache.lucene.index.DocumentsWriter.doFlush() line: 541
org.apache.lucene.index.DocumentsWriter.postUpdate() line: 404
org.apache.lucene.index.DocumentsWriter.updateDocuments() line: 450
org.apache.lucene.index.IndexWriter.updateDocuments() line: 1600
org.apache.lucene.index.IndexWriter.updateDocument() line: 1882
org.apache.lucene.index.IndexWriter.addDocument() line: 1491
perf.IndexThreads$IndexThread.run() line: 419

--- Count: 1 ---
java.io.BufferedOutputStream.growIfNeeded() line: 138
java.io.BufferedOutputStream.write() line: 186
org.apache.lucene.store.OutputStreamIndexOutput.writeBytes() line: 59
org.apache.lucene.store.DataOutput.writeBytes() line: 54
org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter$TermsWriter.writeBlock() line: 944
org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter$TermsWriter.writeBlocks() line: 636
org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter$TermsWriter.pushTerm() line: 1039
org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter$TermsWriter.write() line: 995
org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter.write() line: 390
org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.write() line: 161
org.apache.lucene.index.FreqProxTermsWriter.flush() line: 133
org.apache.lucene.index.IndexingChain.flush() line: 361
org.apache.lucene.index.DocumentsWriterPerThread.flush() line: 492
org.apache.lucene.index.DocumentsWriter.doFlush() line: 541
org.apache.lucene.index.DocumentsWriter.postUpdate() line: 404
org.apache.lucene.index.DocumentsWriter.updateDocuments() line: 450
org.apache.lucene.index.IndexWriter.updateDocuments() line: 1600
org.apache.lucene.index.IndexWriter.updateDocument() line: 1882
org.apache.lucene.index.IndexWriter.addDocument() line: 1491
perf.IndexThreads$IndexThread.run() line: 419

--- Count: 1 ---
java.io.BufferedOutputStream.growIfNeeded() line: 142
java.io.BufferedOutputStream.write() line: 186
org.apache.lucene.store.OutputStreamIndexOutput.writeBytes() line: 59
org.apache.lucene.store.RateLimitedIndexOutput.writeBytes() line: 60
org.apache.lucene.util.bkd.BKDWriter.writeLeafBlockPackedValuesRange() line: 1496
org.apache.lucene.util.bkd.BKDWriter.writeHighCardinalityLeafBlockPackedValues() line: 1422
org.apache.lucene.util.bkd.BKDWriter.writeLeafBlockPackedValues() line: 1358
org.apache.lucene.util.bkd.BKDWriter$OneDimensionBKDWriter.writeLeafBlock() line: 859
org.apache.lucene.util.bkd.BKDWriter$OneDimensionBKDWriter.add() line: 754
org.apache.lucene.util.bkd.BKDWriter.merge() line: 666
org.apache.lucene.codecs.lucene90.Lucene90PointsWriter.merge() line: 279
org.apache.lucene.index.SegmentMerger.mergePoints() line: 197
org.apache.lucene.index.SegmentMerger$$Lambda.0x000000004f1ca8e0.merge() line: -1
org.apache.lucene.index.SegmentMerger.mergeWithLogging() line: 315
org.apache.lucene.index.SegmentMerger.merge() line: 155
org.apache.lucene.index.IndexWriter.mergeMiddle() line: 5322
org.apache.lucene.index.IndexWriter.merge() line: 4785
org.apache.lucene.index.IndexWriter$IndexWriterMergeSource.merge() line: 6590
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge() line: 661
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run() line: 726

--- Count: 1 ---
java.io.BufferedOutputStream.growIfNeeded() line: 138
java.io.BufferedOutputStream.write() line: 186
org.apache.lucene.store.OutputStreamIndexOutput.writeBytes() line: 59
org.apache.lucene.store.RateLimitedIndexOutput.writeBytes() line: 60
org.apache.lucene.store.ByteBuffersDataOutput.copyTo() line: 338
org.apache.lucene.codecs.lucene104.Lucene104PostingsWriter.writeLevel1SkipData() line: 535
org.apache.lucene.codecs.lucene104.Lucene104PostingsWriter.flushDocBlock() line: 487
org.apache.lucene.codecs.lucene104.Lucene104PostingsWriter.startDoc() line: 259
org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm() line: 144
org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter$TermsWriter.write() line: 988
org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter.write() line: 390
org.apache.lucene.codecs.FieldsConsumer.merge() line: 94
org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.merge() line: 190
org.apache.lucene.index.SegmentMerger.mergeTerms() line: 223
org.apache.lucene.index.SegmentMerger$$Lambda.0x000000004f1c48b8.merge() line: -1
org.apache.lucene.index.SegmentMerger.mergeWithLogging() line: 315
org.apache.lucene.index.SegmentMerger.merge() line: 147
org.apache.lucene.index.IndexWriter.mergeMiddle() line: 5322
org.apache.lucene.index.IndexWriter.merge() line: 4785
org.apache.lucene.index.IndexWriter$IndexWriterMergeSource.merge() line: 6590
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge() line: 661
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run() line: 726

--- Count: 1 ---
java.io.BufferedOutputStream.growIfNeeded() line: 134
java.io.BufferedOutputStream.write() line: 186
org.apache.lucene.store.OutputStreamIndexOutput.writeBytes() line: 59
org.apache.lucene.util.bkd.BKDWriter.writeLeafBlockPackedValuesRange() line: 1496
org.apache.lucene.util.bkd.BKDWriter.writeHighCardinalityLeafBlockPackedValues() line: 1422
org.apache.lucene.util.bkd.BKDWriter.writeLeafBlockPackedValues() line: 1358
org.apache.lucene.util.bkd.BKDWriter$OneDimensionBKDWriter.writeLeafBlock() line: 859
org.apache.lucene.util.bkd.BKDWriter$OneDimensionBKDWriter.add() line: 754
org.apache.lucene.util.bkd.BKDWriter$2.visit() line: 609
org.apache.lucene.index.PointValuesWriter$1.visitDocValues() line: 209
org.apache.lucene.util.bkd.BKDWriter.writeField1Dim() line: 604
org.apache.lucene.util.bkd.BKDWriter.writeField() line: 449
org.apache.lucene.codecs.lucene90.Lucene90PointsWriter.writeField() line: 156
org.apache.lucene.index.PointValuesWriter.flush() line: 317
org.apache.lucene.index.IndexingChain.writePoints() line: 408
org.apache.lucene.index.IndexingChain.flush() line: 318
org.apache.lucene.index.DocumentsWriterPerThread.flush() line: 492
org.apache.lucene.index.DocumentsWriter.doFlush() line: 541
org.apache.lucene.index.DocumentsWriter.postUpdate() line: 404
org.apache.lucene.index.DocumentsWriter.updateDocuments() line: 450
org.apache.lucene.index.IndexWriter.updateDocuments() line: 1600
org.apache.lucene.index.IndexWriter.updateDocument() line: 1882
org.apache.lucene.index.IndexWriter.addDocument() line: 1491
perf.IndexThreads$IndexThread.run() line: 419

--- Count: 1 ---
java.io.BufferedOutputStream.growIfNeeded() line: 137
java.io.BufferedOutputStream.write() line: 186
org.apache.lucene.store.OutputStreamIndexOutput.writeBytes() line: 59
org.apache.lucene.store.DataOutput.writeBytes() line: 54
org.apache.lucene.codecs.lucene104.PForUtil.encode() line: 104
org.apache.lucene.codecs.lucene104.Lucene104PostingsWriter.addPosition() line: 342
org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm() line: 159
org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter$TermsWriter.write() line: 988
org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter.write() line: 390
org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.write() line: 161
org.apache.lucene.index.FreqProxTermsWriter.flush() line: 133
org.apache.lucene.index.IndexingChain.flush() line: 361
org.apache.lucene.index.DocumentsWriterPerThread.flush() line: 492
org.apache.lucene.index.DocumentsWriter.doFlush() line: 541
org.apache.lucene.index.DocumentsWriter.postUpdate() line: 404
org.apache.lucene.index.DocumentsWriter.updateDocuments() line: 450
org.apache.lucene.index.IndexWriter.updateDocuments() line: 1600
org.apache.lucene.index.IndexWriter.updateDocument() line: 1882
org.apache.lucene.index.IndexWriter.addDocument() line: 1491
perf.IndexThreads$IndexThread.run() line: 419

--- Count: 1 ---
java.io.BufferedOutputStream.growIfNeeded() line: 134
java.io.BufferedOutputStream.write() line: 186
org.apache.lucene.store.OutputStreamIndexOutput.writeBytes() line: 59
org.apache.lucene.store.DataOutput.writeBytes() line: 54
org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter$TermsWriter.writeBlock() line: 926
org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter$TermsWriter.writeBlocks() line: 666
org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter$TermsWriter.pushTerm() line: 1039
org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter$TermsWriter.write() line: 995
org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter.write() line: 390
org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.write() line: 161
org.apache.lucene.index.FreqProxTermsWriter.flush() line: 133
org.apache.lucene.index.IndexingChain.flush() line: 361
org.apache.lucene.index.DocumentsWriterPerThread.flush() line: 492
org.apache.lucene.index.DocumentsWriter.doFlush() line: 541
org.apache.lucene.index.DocumentsWriter.postUpdate() line: 404
org.apache.lucene.index.DocumentsWriter.updateDocuments() line: 450
org.apache.lucene.index.IndexWriter.updateDocuments() line: 1600
org.apache.lucene.index.IndexWriter.updateDocument() line: 1882
org.apache.lucene.index.IndexWriter.addDocument() line: 1491
perf.IndexThreads$IndexThread.run() line: 419

--- Count: 1 ---
java.io.BufferedOutputStream.growIfNeeded() line: 134
java.io.BufferedOutputStream.write() line: 186
org.apache.lucene.store.OutputStreamIndexOutput.writeBytes() line: 59
org.apache.lucene.store.RateLimitedIndexOutput.writeBytes() line: 60
org.apache.lucene.util.bkd.BKDWriter.writeLeafBlockPackedValuesRange() line: 1496
org.apache.lucene.util.bkd.BKDWriter.writeHighCardinalityLeafBlockPackedValues() line: 1422
org.apache.lucene.util.bkd.BKDWriter.writeLeafBlockPackedValues() line: 1358
org.apache.lucene.util.bkd.BKDWriter$OneDimensionBKDWriter.writeLeafBlock() line: 859
org.apache.lucene.util.bkd.BKDWriter$OneDimensionBKDWriter.add() line: 754
org.apache.lucene.util.bkd.BKDWriter.merge() line: 666
org.apache.lucene.codecs.lucene90.Lucene90PointsWriter.merge() line: 279
org.apache.lucene.index.SegmentMerger.mergePoints() line: 197
org.apache.lucene.index.SegmentMerger$$Lambda.0x000000004f1ca8e0.merge() line: -1
org.apache.lucene.index.SegmentMerger.mergeWithLogging() line: 315
org.apache.lucene.index.SegmentMerger.merge() line: 155
org.apache.lucene.index.IndexWriter.mergeMiddle() line: 5322
org.apache.lucene.index.IndexWriter.merge() line: 4785
org.apache.lucene.index.IndexWriter$IndexWriterMergeSource.merge() line: 6590
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge() line: 661
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run() line: 726

--- Count: 1 ---
java.io.BufferedOutputStream.growIfNeeded() line: 142
java.io.BufferedOutputStream.write() line: 186
org.apache.lucene.store.OutputStreamIndexOutput.writeBytes() line: 59
org.apache.lucene.store.ByteBuffersDataOutput.copyTo() line: 338
org.apache.lucene.codecs.lucene104.Lucene104PostingsWriter.writeLevel1SkipData() line: 535
org.apache.lucene.codecs.lucene104.Lucene104PostingsWriter.flushDocBlock() line: 487
org.apache.lucene.codecs.lucene104.Lucene104PostingsWriter.startDoc() line: 259
org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm() line: 144
org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter$TermsWriter.write() line: 988
org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter.write() line: 390
org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.write() line: 161
org.apache.lucene.index.FreqProxTermsWriter.flush() line: 133
org.apache.lucene.index.IndexingChain.flush() line: 361
org.apache.lucene.index.DocumentsWriterPerThread.flush() line: 492
org.apache.lucene.index.DocumentsWriter.doFlush() line: 541
org.apache.lucene.index.DocumentsWriter.postUpdate() line: 404
org.apache.lucene.index.DocumentsWriter.updateDocuments() line: 450
org.apache.lucene.index.IndexWriter.updateDocuments() line: 1600
org.apache.lucene.index.IndexWriter.updateDocument() line: 1882
org.apache.lucene.index.IndexWriter.addDocument() line: 1491
perf.IndexThreads$IndexThread.run() line: 419

--- Count: 1 ---
java.io.BufferedOutputStream.growIfNeeded() line: 134
java.io.BufferedOutputStream.write() line: 186
org.apache.lucene.store.OutputStreamIndexOutput.writeBytes() line: 59
org.apache.lucene.store.ByteBuffersDataOutput.copyTo() line: 338
org.apache.lucene.codecs.lucene90.LZ4WithPresetDictCompressionMode$LZ4WithPresetDictCompressor.compress() line: 194
org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsWriter.flush() line: 259
org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsWriter.finishDocument() line: 191
org.apache.lucene.index.StoredFieldsConsumer.finishDocument() line: 94
org.apache.lucene.index.IndexingChain.finishStoredFields() line: 565
org.apache.lucene.index.IndexingChain.processDocument() line: 657
org.apache.lucene.index.DocumentsWriterPerThread.updateDocuments() line: 291
org.apache.lucene.index.DocumentsWriter.updateDocuments() line: 428
org.apache.lucene.index.IndexWriter.updateDocuments() line: 1600
org.apache.lucene.index.IndexWriter.updateDocument() line: 1882
org.apache.lucene.index.IndexWriter.addDocument() line: 1491
perf.IndexThreads$IndexThread.run() line: 419

--- Count: 1 ---
java.io.BufferedOutputStream.growIfNeeded() line: 137
java.io.BufferedOutputStream.write() line: 186
org.apache.lucene.store.OutputStreamIndexOutput.writeBytes() line: 59
org.apache.lucene.util.bkd.BKDWriter.writeLeafBlockPackedValuesRange() line: 1496
org.apache.lucene.util.bkd.BKDWriter.writeHighCardinalityLeafBlockPackedValues() line: 1422
org.apache.lucene.util.bkd.BKDWriter.writeLeafBlockPackedValues() line: 1358
org.apache.lucene.util.bkd.BKDWriter$OneDimensionBKDWriter.writeLeafBlock() line: 859
org.apache.lucene.util.bkd.BKDWriter$OneDimensionBKDWriter.add() line: 754
org.apache.lucene.util.bkd.BKDWriter$2.visit() line: 609
org.apache.lucene.index.PointValuesWriter$1.visitDocValues() line: 209
org.apache.lucene.util.bkd.BKDWriter.writeField1Dim() line: 604
org.apache.lucene.util.bkd.BKDWriter.writeField() line: 449
org.apache.lucene.codecs.lucene90.Lucene90PointsWriter.writeField() line: 156
org.apache.lucene.index.PointValuesWriter.flush() line: 317
org.apache.lucene.index.IndexingChain.writePoints() line: 408
org.apache.lucene.index.IndexingChain.flush() line: 318
org.apache.lucene.index.DocumentsWriterPerThread.flush() line: 492
org.apache.lucene.index.DocumentsWriter.doFlush() line: 541
org.apache.lucene.index.DocumentsWriter.postUpdate() line: 404
org.apache.lucene.index.DocumentsWriter.updateDocuments() line: 450
org.apache.lucene.index.IndexWriter.updateDocuments() line: 1600
org.apache.lucene.index.IndexWriter.updateDocument() line: 1882
org.apache.lucene.index.IndexWriter.addDocument() line: 1491
perf.IndexThreads$IndexThread.run() line: 419

--- Count: 1 ---
java.io.BufferedOutputStream.growIfNeeded() line: 135
java.io.BufferedOutputStream.write() line: 186
org.apache.lucene.store.OutputStreamIndexOutput.writeBytes() line: 59
org.apache.lucene.store.RateLimitedIndexOutput.writeBytes() line: 60
org.apache.lucene.store.DataOutput.writeBytes() line: 54
org.apache.lucene.codecs.lucene104.PForUtil.encode() line: 104
org.apache.lucene.codecs.lucene104.Lucene104PostingsWriter.addPosition() line: 342
org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm() line: 159
org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter$TermsWriter.write() line: 988
org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter.write() line: 390
org.apache.lucene.codecs.FieldsConsumer.merge() line: 94
org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.merge() line: 190
org.apache.lucene.index.SegmentMerger.mergeTerms() line: 223
org.apache.lucene.index.SegmentMerger$$Lambda.0x000000004f1c48b8.merge() line: -1
org.apache.lucene.index.SegmentMerger.mergeWithLogging() line: 315
org.apache.lucene.index.SegmentMerger.merge() line: 147
org.apache.lucene.index.IndexWriter.mergeMiddle() line: 5322
org.apache.lucene.index.IndexWriter.merge() line: 4785
org.apache.lucene.index.IndexWriter$IndexWriterMergeSource.merge() line: 6590
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge() line: 661
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run() line: 726

Total CPU samples: 409250
Matched: 26
Percentage: 0.01%

Baseline top stacks of knn 1MM

--- Count: 3 ---
java.io.BufferedOutputStream.growIfNeeded() line: 134
java.io.BufferedOutputStream.write() line: 186
org.apache.lucene.store.OutputStreamIndexOutput.writeBytes() line: 59
org.apache.lucene.store.RateLimitedIndexOutput.writeBytes() line: 60
org.apache.lucene.store.DataOutput.writeBytes() line: 54
org.apache.lucene.util.GroupVIntUtil.writeGroupVInts() line: 173
org.apache.lucene.store.DataOutput.writeGroupVInts() line: 358
org.apache.lucene.codecs.lucene99.Lucene99HnswVectorsWriter.writeGraph() line: 550
org.apache.lucene.codecs.lucene99.Lucene99HnswVectorsWriter.buildAndWriteGraph() line: 484
org.apache.lucene.codecs.lucene99.Lucene99HnswVectorsWriter.lambda$mergeOneField$0() line: 451
org.apache.lucene.codecs.lucene99.Lucene99HnswVectorsWriter$$Lambda.0x000000003019a9e0.run() line: -1
org.apache.lucene.codecs.KnnVectorsWriter.merge() line: 144
org.apache.lucene.index.SegmentMerger.mergeVectorValues() line: 272
org.apache.lucene.index.SegmentMerger$$Lambda.0x0000000030198000.merge() line: -1
org.apache.lucene.index.SegmentMerger.mergeWithLogging() line: 315
org.apache.lucene.index.SegmentMerger.merge() line: 159
org.apache.lucene.index.IndexWriter.mergeMiddle() line: 5322
org.apache.lucene.index.IndexWriter.merge() line: 4785
org.apache.lucene.index.IndexWriter$IndexWriterMergeSource.merge() line: 6590
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge() line: 661
knn.KnnIndexer$TrackingConcurrentMergeScheduler.doMerge() line: 360
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run() line: 726

--- Count: 1 ---
java.io.BufferedOutputStream.growIfNeeded() line: 137
java.io.BufferedOutputStream.write() line: 186
org.apache.lucene.store.OutputStreamIndexOutput.writeBytes() line: 59
org.apache.lucene.store.RateLimitedIndexOutput.writeBytes() line: 60
org.apache.lucene.store.DataOutput.writeBytes() line: 54
org.apache.lucene.util.GroupVIntUtil.writeGroupVInts() line: 173
org.apache.lucene.store.DataOutput.writeGroupVInts() line: 358
org.apache.lucene.codecs.lucene99.Lucene99HnswVectorsWriter.writeGraph() line: 550
org.apache.lucene.codecs.lucene99.Lucene99HnswVectorsWriter.buildAndWriteGraph() line: 484
org.apache.lucene.codecs.lucene99.Lucene99HnswVectorsWriter.lambda$mergeOneField$0() line: 451
org.apache.lucene.codecs.lucene99.Lucene99HnswVectorsWriter$$Lambda.0x000000003019a9e0.run() line: -1
org.apache.lucene.codecs.KnnVectorsWriter.merge() line: 144
org.apache.lucene.index.SegmentMerger.mergeVectorValues() line: 272
org.apache.lucene.index.SegmentMerger$$Lambda.0x0000000030198000.merge() line: -1
org.apache.lucene.index.SegmentMerger.mergeWithLogging() line: 315
org.apache.lucene.index.SegmentMerger.merge() line: 159
org.apache.lucene.index.IndexWriter.mergeMiddle() line: 5322
org.apache.lucene.index.IndexWriter.merge() line: 4785
org.apache.lucene.index.IndexWriter$IndexWriterMergeSource.merge() line: 6590
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge() line: 661
knn.KnnIndexer$TrackingConcurrentMergeScheduler.doMerge() line: 360
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run() line: 726

Total CPU samples: 1943015
Matched: 4
Percentage: 0.00%

@github-actions github-actions Bot added this to the 10.5.0 milestone May 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

module:core/store

Projects

None yet

Milestone

10.5.0

Development

Successfully merging this pull request may close these issues.

1 participant

@neoremind