Low-overhead profiling #77

rmuir · 2020-07-31T23:39:59Z

I think the benchmark results would be more useful if we could see some basic analysis of the code under load. We could imagine summary tables, graphs, flame charts, whatever is needed to knock out the hotspots and keep them under control.

Instead of seeing just that performance dropped or improved, we might be able to reason about why: it could make things better for developers. And maybe, just maybe, we have some giant frogs boiling for years that nobody never noticed yet: you know, easy wins to improve perf that went undetected.

As a start I imagine breaking the problem down into two profiling mechamisms:

"system profiler": linux perf equivalent. We can "teach" it java with some JVM flags + agent. See blog post: https://jvns.ca/blog/2016/02/24/perf-top-my-new-best-friend/. I think in this case it is a matter of downloading the agent and tweaking the JVM args, wrapping with correct perf commandlines. We don't have to start with a fancy flamegraph report, it could be a simple table to get things going. I really like how this tools "drills down" even into the kernel to find low-level bottlenecks. Maybe dtrace support on the mac could do the equivalent for local runs, and make it easier on developers.
java profiling: just using java flight recorder, similar to lucene's test suite. There we just apply some flags and collect the data files correctly across different jvms/projects (it is convenient that the default filename includes pid for the multi-jvm case!): https://github.com/apache/lucene-solr/blob/master/gradle/testing/profiling.gradle . Hopefully the logic is easy to see: we just clean the relevant recordings beforehand and collect the relevant ones after and feed them to a reporter (https://github.com/apache/lucene-solr/blob/master/buildSrc/src/main/java/org/apache/lucene/gradle/ProfileResults.java). To exactly specify the overhead of the metrics we collect, we supply a configuration file: https://github.com/apache/lucene-solr/blob/master/gradle/testing/profiling.jfc

rmuir · 2020-11-29T03:00:07Z

linux perf usage becomes a lot easier with jdk-16 early access build. We don't need to compile/install any java agent and can just pass JVM flags such as -XX:+UnlockDiagnosticVMOptions -XX:+DumpPerfMapAtExit -XX:+PreserveFramePointer, and run the benchmark with perf record. See https://bugs.openjdk.java.net/browse/JDK-8254723

mikemccand · 2021-01-05T23:27:18Z

And maybe, just maybe, we have some giant frogs boiling for years that nobody never noticed yet

LOL

mikemccand · 2021-01-05T23:35:07Z

I chatted w/ @rmuir and he suggested we start simpler here with Java Flight Recorder (JFR). We can later upgrade to perf. Progress not perfection!

Some notes from our chat:

We can maybe use existing simple Lucene command-line tool to merge multiple runs together (since search benchmarks run 20 JVM iterations): https://github.com/apache/lucene-solr/blob/master/buildSrc/src/main/java/org/apache/lucene/gradle/ProfileResults.java -- run this once for CPU results, and again for heap allocations aggregation
As an example of the right JVM command-line flags, this is what Lucene's build infra does when you pass -Dtests.profile: https://github.com/apache/lucene-solr/blob/master/gradle/testing/profiling.gradle
And here is the custom flight recorder config for -Dtests.profile: https://github.com/apache/lucene-solr/blob/master/gradle/testing/profiling.jfc
Be sure to remove *.jfr before each run!

mikemccand · 2021-01-10T18:32:42Z

I have a start at this! Still iterating on it ... it produces output like this, after all 20 JVM iterations of a benchmark run, when running on wikimediumall:

CPU merged search profile for comp:
PROFILE SUMMARY from 1060297 events (total: 1M)
  tests.profile.mode=cpu
  tests.profile.count=30
  tests.profile.stacksize=1
  tests.profile.linenumbers=false
PERCENT       CPU SAMPLES   STACK
8.32%         88215         org.apache.lucene.util.packed.DirectReader$DirectPackedReader12#get()
8.03%         85123         org.apache.lucene.util.packed.DirectMonotonicReader#get()
6.41%         67966         org.apache.lucene.facet.taxonomy.FastTaxonomyFacetCounts#countAll()
6.14%         65140         org.apache.lucene.codecs.lucene80.Lucene80DocValuesProducer$15#binaryValue()
5.21%         55271         org.apache.lucene.facet.sortedset.SortedSetDocValuesFacetCounts#countOneSegment()
2.88%         30504         org.apache.lucene.facet.taxonomy.IntTaxonomyFacets#increment()
2.86%         30302         java.nio.DirectByteBuffer#get()
2.72%         28800         jdk.internal.misc.Unsafe#convEndian()
2.14%         22713         org.apache.lucene.store.ByteBufferGuard#ensureValid()
2.07%         21976         java.nio.Buffer#checkIndex()
1.81%         19224         java.nio.ByteBuffer#get()
1.56%         16575         java.nio.Buffer#position()
1.45%         15367         org.apache.lucene.util.packed.DirectReader$DirectPackedReader4#get()
1.20%         12770         java.nio.Buffer#checkSegment()
1.20%         12756         org.apache.lucene.store.ByteBufferGuard#getBytes()
1.12%         11868         org.apache.lucene.store.ByteBufferIndexInput#setCurBuf()
1.02%         10787         java.nio.Buffer#nextGetIndex()
1.00%         10633         org.apache.lucene.search.Weight$DefaultBulkScorer#scoreAll()
0.96%         10209         org.apache.lucene.search.PhraseScorer$1#matches()
0.94%         9916          org.apache.lucene.store.ByteBufferIndexInput$SingleBufferImpl#seek()
0.93%         9836          org.apache.lucene.codecs.lucene84.Lucene84PostingsReader$EverythingEnum#nextPosition()
0.83%         8758          org.apache.lucene.index.SingletonSortedSetDocValues#nextOrd()
0.82%         8688          org.apache.lucene.codecs.lucene80.Lucene80NormsProducer$3#longValue()
0.80%         8465          jdk.internal.misc.Unsafe#copyMemory()
0.77%         8188          org.apache.lucene.search.ConjunctionDISI#doNext()
0.74%         7833          org.apache.lucene.codecs.lucene80.Lucene80DocValuesProducer$DenseBinaryDocValues#advance()
0.71%         7481          org.apache.lucene.util.PriorityQueue#add()
0.66%         6981          org.apache.lucene.codecs.lucene84.Lucene84PostingsReader$EverythingEnum#skipPositions()
0.65%         6865          org.apache.lucene.search.spans.TermSpans#nextStartPosition()
0.63%         6682          org.apache.lucene.search.spans.NearSpansOrdered#stretchToOrder()

HEAP merged search profile for comp:
PROFILE SUMMARY from 28436 events (total: 25291M)
  tests.profile.mode=heap
  tests.profile.count=30
  tests.profile.stacksize=1
  tests.profile.linenumbers=false
PERCENT       HEAP SAMPLES  STACK
13.62%        3445M         org.apache.lucene.util.FixedBitSet#<init>()
6.46%         1634M         org.apache.lucene.codecs.blocktree.SegmentTermsEnumFrame#<init>()
6.16%         1557M         java.util.AbstractList#iterator()
5.14%         1299M         org.apache.lucene.util.BytesRef#<init>()
5.12%         1294M         org.apache.lucene.util.ArrayUtil#growExact()
4.15%         1048M         org.apache.lucene.util.fst.ByteSequenceOutputs#read()
4.01%         1014M         org.apache.lucene.queryparser.charstream.FastCharStream#refill()
2.84%         719M          org.apache.lucene.codecs.lucene84.Lucene84PostingsReader$BlockDocsEnum#<init>()
2.78%         702M          org.apache.lucene.search.ExactPhraseMatcher$1$1#getImpacts()
2.37%         598M          org.apache.lucene.search.ExactPhraseMatcher$1#getImpacts()
2.00%         506M          org.apache.lucene.util.DocIdSetBuilder$Buffer#<init>()
1.86%         470M          java.nio.DirectByteBufferR#duplicate()
1.85%         468M          jdk.internal.misc.Unsafe#allocateUninitializedArray()
1.85%         467M          org.apache.lucene.store.DataInput#skipBytes()
1.83%         463M          org.apache.lucene.codecs.blocktree.IntersectTermsEnumFrame#load()
1.55%         392M          org.apache.lucene.codecs.blocktree.SegmentTermsEnum#getFrame()
1.37%         347M          org.apache.lucene.codecs.lucene84.Lucene84PostingsReader$BlockImpactsDocsEnum#<init>()
1.27%         320M          org.apache.lucene.codecs.lucene84.Lucene84PostingsReader#newTermState()
1.23%         312M          java.util.ArrayList#grow()
1.22%         308M          org.apache.lucene.codecs.lucene84.ForUtil#<init>()
1.18%         298M          java.util.ArrayList#iterator()
1.18%         298M          org.apache.lucene.codecs.blocktree.SegmentTermsEnum#<init>()
1.07%         271M          org.apache.lucene.store.ByteBufferIndexInput#buildSlice()
1.07%         270M          org.apache.lucene.codecs.blocktree.IntersectTermsEnumFrame#<init>()
0.93%         236M          java.nio.DirectByteBufferR#asLongBuffer()
0.90%         226M          org.apache.lucene.codecs.lucene84.Lucene84PostingsReader$BlockImpactsPostingsEnum#<init>()
0.89%         224M          org.apache.lucene.store.ByteBufferIndexInput#newCloneInstance()
0.80%         201M          java.util.Arrays#copyOf()
0.78%         196M          java.nio.DirectByteBufferR#slice()
0.76%         191M          org.apache.lucene.codecs.blocktree.SegmentTermsEnum#getArc()

Look at that 2.72% CPU in convEndian and 2.15% in ByteBufferGuard#ensureValid!

…rks and in nightly search benchmarks

mikemccand · 2021-01-12T12:22:59Z

I pushed JFR for searching (still need to enable for indexing too), and it ran last night for the first time: https://home.apache.org/~mikemccand/lucenebench/2021.01.12.00.03.16.html

The results are ... curious. E.g. 39% of HEAP samples were for java.lang.Integer#valueOf()!? Are we really boxing that many integers (where!).

6.4% of HEAP was jdk.internal.math.FloatingDecimal#readJavaFormatString()!? I think this is parsing String to float/double, sheesh.

~2% of CPU is spent in java.nio.ByteBuffer#get() -- are we really pulling single bytes so often!?

I will increase reported stack depth so we can get more details :)

mikemccand · 2021-01-12T13:30:22Z

OK I re-ran the *.jfr aggregation, asking for stacksize=10. I think we really must fix our grouping implementation to not box/unbox Integer on every hit!

PERCENT       HEAP SAMPLES  STACK
39.19%        330167M       java.lang.Integer#valueOf()
                              at org.apache.lucene.search.grouping.TermGroupSelector#advanceTo()
                              at org.apache.lucene.search.grouping.SecondPassGroupingCollector#collect()
                              at org.apache.lucene.search.Weight$DefaultBulkScorer#scoreAll()
                              at org.apache.lucene.search.Weight$DefaultBulkScorer#score()
                              at org.apache.lucene.search.BulkScorer#score()
                              at org.apache.lucene.search.IndexSearcher#search()
                              at org.apache.lucene.search.IndexSearcher#search()
                              at perf.SearchTask#go()
                              at perf.TaskThreads$TaskThread#run()
6.38%         53776M        jdk.internal.math.FloatingDecimal#readJavaFormatString()
                              at jdk.internal.math.FloatingDecimal#parseFloat()
                              at java.lang.Float#parseFloat()
                              at perf.VectorDictionary#parseLine()
                              at perf.VectorDictionary#<init>()
                              at perf.TaskParser#<init>()
                              at perf.SearchPerfTest#_main()
                              at perf.SearchPerfTest#main()
4.00%         33743M        org.apache.lucene.util.FixedBitSet#<init>()
                              at org.apache.lucene.util.DocIdSetBuilder#upgradeToBitSet()
                              at org.apache.lucene.util.DocIdSetBuilder#grow()
                              at org.apache.lucene.util.DocIdSetBuilder#add()
                              at org.apache.lucene.search.MultiTermQueryConstantScoreWrapper$1#rewrite()
                              at org.apache.lucene.search.MultiTermQueryConstantScoreWrapper$1#bulkScorer()
                              at org.apache.lucene.search.IndexSearcher#search()
                              at org.apache.lucene.search.IndexSearcher#search()
                              at org.apache.lucene.search.IndexSearcher#search()
                              at org.apache.lucene.search.IndexSearcher#searchAfter()
3.16%         26630M        java.lang.StringLatin1#newString()
                              at java.lang.String#substring()
                              at java.lang.String#split()
                              at java.lang.String#split()
                              at perf.VectorDictionary#parseLine()
                              at perf.VectorDictionary#<init>()
                              at perf.TaskParser#<init>()
                              at perf.SearchPerfTest#_main()
                              at perf.SearchPerfTest#main()
2.39%         20101M        java.util.Arrays#copyOf()
                              at java.util.ArrayList#grow()
                              at java.util.ArrayList#grow()
                              at java.util.ArrayList#add()
                              at java.util.ArrayList#add()
                              at java.lang.String#split()
                              at java.lang.String#split()
                              at perf.VectorDictionary#parseLine()
                              at perf.VectorDictionary#<init>()
                              at perf.TaskParser#<init>()
2.28%         19242M        java.util.Arrays#copyOfRange()
                              at java.lang.StringLatin1#newString()
                              at java.lang.String#substring()
                              at java.lang.String#split()
                              at java.lang.String#split()
                              at perf.VectorDictionary#parseLine()
                              at perf.VectorDictionary#<init>()
                              at perf.TaskParser#<init>()
                              at perf.SearchPerfTest#_main()
                              at perf.SearchPerfTest#main()
1.88%         15839M        org.apache.lucene.util.FixedBitSet#<init>()
                              at org.apache.lucene.util.DocIdSetBuilder#upgradeToBitSet()
                              at org.apache.lucene.util.DocIdSetBuilder#grow()
                              at org.apache.lucene.facet.FacetsCollector#collect()
                              at org.apache.lucene.search.MultiCollector$MultiLeafCollector#collect()
                              at org.apache.lucene.search.Weight$DefaultBulkScorer#scoreAll()
                              at org.apache.lucene.search.Weight$DefaultBulkScorer#score()
                              at org.apache.lucene.search.BulkScorer#score()
                              at org.apache.lucene.search.IndexSearcher#search()
                              at org.apache.lucene.search.IndexSearcher#search()
1.73%         14598M        java.lang.ref.Finalizer#runFinalization()
                              at java.lang.ref.Reference$1#runFinalization()
                              at java.lang.Runtime#runFinalization()
                              at perf.SearchPerfTest#_main()
                              at perf.SearchPerfTest#main()
1.72%         14517M        org.apache.lucene.search.ExactPhraseMatcher$1$1#getImpacts()
                              at org.apache.lucene.search.MaxScoreCache#getMaxScoreForLevel()
                              at org.apache.lucene.search.ImpactsDISI#advanceTarget()
                              at org.apache.lucene.search.ImpactsDISI#advance()
                              at org.apache.lucene.search.ImpactsDISI#nextDoc()
                              at org.apache.lucene.search.Weight$DefaultBulkScorer#scoreAll()
                              at org.apache.lucene.search.Weight$DefaultBulkScorer#score()
                              at org.apache.lucene.search.BulkScorer#score()
                              at org.apache.lucene.search.IndexSearcher#search()
                              at org.apache.lucene.search.IndexSearcher#search()
1.26%         10575M        java.util.AbstractList#listIterator()
                              at java.util.AbstractList$SubList$1#<init>()
                              at java.util.AbstractList$SubList#listIterator()
                              at java.util.AbstractList#listIterator()
                              at java.util.AbstractList$SubList#iterator()
                              at org.apache.lucene.search.DisiPriorityQueue#iterator()
                              at org.apache.lucene.search.WANDScorer#updateMaxScores()
                              at org.apache.lucene.search.WANDScorer#updateMaxScoresIfNecessary()
                              at org.apache.lucene.search.WANDScorer#moveToNextCandidate()
                              at org.apache.lucene.search.WANDScorer$1#advance()
0.90%         7559M         java.lang.String#split()
                              at java.lang.String#split()
                              at perf.VectorDictionary#parseLine()
                              at perf.VectorDictionary#<init>()
                              at perf.TaskParser#<init>()
                              at perf.SearchPerfTest#_main()
                              at perf.SearchPerfTest#main()
0.87%         7371M         java.util.AbstractList#iterator()
                              at org.apache.lucene.search.MaxScoreCache#computeMaxScore()
                              at org.apache.lucene.search.MaxScoreCache#getMaxScoreForLevel()
                              at org.apache.lucene.search.ImpactsDISI#advanceTarget()
                              at org.apache.lucene.search.ImpactsDISI#advance()
                              at org.apache.lucene.search.WANDScorer#advanceTail()
                              at org.apache.lucene.search.WANDScorer#advanceTail()
                              at org.apache.lucene.search.WANDScorer$2#matches()
                              at org.apache.lucene.search.Weight$DefaultBulkScorer#scoreAll()
                              at org.apache.lucene.search.Weight$DefaultBulkScorer#score()
0.85%         7163M         org.apache.lucene.search.ExactPhraseMatcher$1#getImpacts()
                              at org.apache.lucene.search.MaxScoreCache#getMaxScoreForLevel()
                              at org.apache.lucene.search.MaxScoreCache#getSkipLevel()
                              at org.apache.lucene.search.MaxScoreCache#getSkipUpTo()
                              at org.apache.lucene.search.ImpactsDISI#advanceTarget()
                              at org.apache.lucene.search.ImpactsDISI#advance()
                              at org.apache.lucene.search.ImpactsDISI#nextDoc()
                              at org.apache.lucene.search.Weight$DefaultBulkScorer#scoreAll()
                              at org.apache.lucene.search.Weight$DefaultBulkScorer#score()
                              at org.apache.lucene.search.BulkScorer#score()
0.82%         6901M         org.apache.lucene.util.FixedBitSet#<init>()
                              at org.apache.lucene.search.PointRangeQuery$1$4#get()
                              at org.apache.lucene.search.PointRangeQuery$1#scorer()
                              at org.apache.lucene.search.Weight#bulkScorer()
                              at org.apache.lucene.search.IndexSearcher#search()
                              at org.apache.lucene.search.IndexSearcher#search()
                              at org.apache.lucene.search.IndexSearcher#search()
                              at org.apache.lucene.search.IndexSearcher#searchAfter()
                              at org.apache.lucene.search.IndexSearcher#search()
                              at perf.SearchTask#go()
0.78%         6565M         org.apache.lucene.util.FixedBitSet#<init>()
                              at org.apache.lucene.util.DocIdSetBuilder#upgradeToBitSet()
                              at org.apache.lucene.util.DocIdSetBuilder#grow()
                              at org.apache.lucene.search.PointRangeQuery$1$1#grow()
                              at org.apache.lucene.util.bkd.BKDReader#addAll()
                              at org.apache.lucene.util.bkd.BKDReader#intersect()
                              at org.apache.lucene.util.bkd.BKDReader#intersect()
                              at org.apache.lucene.util.bkd.BKDReader#intersect()
                              at org.apache.lucene.util.bkd.BKDReader#intersect()
                              at org.apache.lucene.util.bkd.BKDReader#intersect()
0.75%         6356M         java.lang.StringUTF16#compress()
                              at java.lang.String#<init>()
                              at java.lang.String#<init>()
                              at java.io.BufferedReader#readLine()
                              at java.io.BufferedReader#readLine()
                              at perf.VectorDictionary#<init>()
                              at perf.TaskParser#<init>()
                              at perf.SearchPerfTest#_main()
                              at perf.SearchPerfTest#main()
0.71%         6003M         java.util.AbstractList#iterator()
                              at org.apache.lucene.search.MaxScoreCache#computeMaxScore()
                              at org.apache.lucene.search.MaxScoreCache#getMaxScoreForLevel()
                              at org.apache.lucene.search.ImpactsDISI#getMaxScore()
                              at org.apache.lucene.search.TermScorer#getMaxScore()
                              at org.apache.lucene.search.WANDScorer#updateMaxScores()
                              at org.apache.lucene.search.WANDScorer#updateMaxScoresIfNecessary()
                              at org.apache.lucene.search.WANDScorer#moveToNextCandidate()
                              at org.apache.lucene.search.WANDScorer$1#advance()
                              at org.apache.lucene.search.BlockMaxConjunctionScorer$2#doNext()
0.66%         5520M         java.util.ArrayList#grow()
                              at java.util.ArrayList#grow()
                              at java.util.ArrayList#add()
                              at java.util.ArrayList#add()
                              at org.apache.lucene.search.ExactPhraseMatcher$1$1#getImpacts()
                              at org.apache.lucene.search.MaxScoreCache#getMaxScoreForLevel()
                              at org.apache.lucene.search.ImpactsDISI#advanceTarget()
                              at org.apache.lucene.search.ImpactsDISI#advance()
                              at org.apache.lucene.search.ImpactsDISI#nextDoc()
                              at org.apache.lucene.search.Weight$DefaultBulkScorer#scoreAll()
0.61%         5154M         java.util.concurrent.CopyOnWriteArrayList#iterator()
                              at jdk.jfr.internal.RequestEngine#run_requests()
                              at jdk.jfr.internal.RequestEngine#doPeriodic()
                              at jdk.jfr.internal.PlatformRecorder#periodicTask()
                              at jdk.jfr.internal.PlatformRecorder#lambda$startDiskMonitor$1()
                              at jdk.jfr.internal.PlatformRecorder$$Lambda$44+0x0000000800b74740.1164107853#run()
                              at java.lang.Thread#run()
0.58%         4906M         org.apache.lucene.codecs.lucene84.Lucene84PostingsReader$BlockDocsEnum#<init>()
                              at org.apache.lucene.codecs.lucene84.Lucene84PostingsReader#postings()
                              at org.apache.lucene.codecs.lucene84.Lucene84PostingsReader#impacts()
                              at org.apache.lucene.codecs.blocktree.SegmentTermsEnum#impacts()
                              at org.apache.lucene.search.TermQuery$TermWeight#scorer()
                              at org.apache.lucene.search.Weight#scorerSupplier()
                              at org.apache.lucene.search.BooleanWeight#scorerSupplier()
                              at org.apache.lucene.search.BooleanWeight#scorer()
                              at org.apache.lucene.search.Weight#bulkScorer()
                              at org.apache.lucene.search.BooleanWeight#bulkScorer()
0.54%         4514M         java.util.Arrays#asList()
                              at org.apache.lucene.search.DisiPriorityQueue#iterator()
                              at org.apache.lucene.search.WANDScorer#updateMaxScores()
                              at org.apache.lucene.search.WANDScorer#updateMaxScoresIfNecessary()
                              at org.apache.lucene.search.WANDScorer#moveToNextCandidate()
                              at org.apache.lucene.search.WANDScorer$1#advance()
                              at org.apache.lucene.search.BlockMaxConjunctionScorer$2#doNext()
                              at org.apache.lucene.search.BlockMaxConjunctionScorer$2#advance()
                              at org.apache.lucene.search.BlockMaxConjunctionScorer$2#nextDoc()
                              at org.apache.lucene.search.Weight$DefaultBulkScorer#scoreAll()
0.53%         4494M         java.util.AbstractList#iterator()
                              at org.apache.lucene.search.MaxScoreCache#computeMaxScore()
                              at org.apache.lucene.search.MaxScoreCache#getMaxScoreForLevel()
                              at org.apache.lucene.search.ImpactsDISI#getMaxScore()
                              at org.apache.lucene.search.TermScorer#getMaxScore()
                              at org.apache.lucene.search.BlockMaxConjunctionScorer#getMaxScore()
                              at org.apache.lucene.search.BlockMaxConjunctionScorer$2#moveToNextBlock()
                              at org.apache.lucene.search.BlockMaxConjunctionScorer$2#advanceTarget()
                              at org.apache.lucene.search.BlockMaxConjunctionScorer$2#doNext()
                              at org.apache.lucene.search.BlockMaxConjunctionScorer$2#advance()
0.52%         4416M         java.util.AbstractList#iterator()
                              at org.apache.lucene.search.MaxScoreCache#computeMaxScore()
                              at org.apache.lucene.search.MaxScoreCache#getMaxScoreForLevel()
                              at org.apache.lucene.search.ImpactsDISI#advanceTarget()
                              at org.apache.lucene.search.ImpactsDISI#advance()
                              at org.apache.lucene.search.BlockMaxConjunctionScorer$2#doNext()
                              at org.apache.lucene.search.BlockMaxConjunctionScorer$2#advance()
                              at org.apache.lucene.search.BlockMaxConjunctionScorer$2#nextDoc()
                              at org.apache.lucene.search.Weight$DefaultBulkScorer#scoreAll()
                              at org.apache.lucene.search.Weight$DefaultBulkScorer#score()
0.51%         4295M         perf.VectorDictionary#parseLine()
                              at perf.VectorDictionary#<init>()
                              at perf.TaskParser#<init>()
                              at perf.SearchPerfTest#_main()
                              at perf.SearchPerfTest#main()
0.50%         4241M         jdk.internal.misc.Unsafe#allocateUninitializedArray()
                              at java.lang.StringConcatHelper#newArray()
                              at java.lang.StringConcatHelper#newArrayWithSuffix()
                              at java.lang.invoke.DirectMethodHandle$Holder#invokeStatic()
                              at java.lang.invoke.LambdaForm$MH+0x0000000800bb3040.1462044018#invoke()
                              at java.lang.invoke.LambdaForm$MH+0x0000000800ba6440.1536471117#invoke()
                              at java.lang.invoke.Invokers$Holder#linkToTargetMethod()
                              at org.apache.lucene.store.IndexInput#getFullSliceDescription()
                              at org.apache.lucene.store.ByteBufferIndexInput#buildSlice()
                              at org.apache.lucene.store.ByteBufferIndexInput#slice()
0.50%         4186M         org.apache.lucene.search.ExactPhraseMatcher$1#getImpacts()
                              at org.apache.lucene.search.MaxScoreCache#getMaxScoreForLevel()
                              at org.apache.lucene.search.ImpactsDISI#advanceTarget()
                              at org.apache.lucene.search.ImpactsDISI#advance()
                              at org.apache.lucene.search.ImpactsDISI#nextDoc()
                              at org.apache.lucene.search.Weight$DefaultBulkScorer#scoreAll()
                              at org.apache.lucene.search.Weight$DefaultBulkScorer#score()
                              at org.apache.lucene.search.BulkScorer#score()
                              at org.apache.lucene.search.IndexSearcher#search()
                              at org.apache.lucene.search.IndexSearcher#search()
0.50%         4180M         java.util.concurrent.locks.AbstractQueuedSynchronizer#acquire()
                              at java.util.concurrent.locks.AbstractQueuedSynchronizer#acquireSharedInterruptibly()
                              at java.util.concurrent.CountDownLatch#await()
                              at perf.TaskThreads$TaskThread#run()
0.49%         4099M         org.apache.lucene.util.DocIdSetBuilder$Buffer#<init>()
                              at org.apache.lucene.util.DocIdSetBuilder#addBuffer()
                              at org.apache.lucene.util.DocIdSetBuilder#ensureBufferCapacity()
                              at org.apache.lucene.util.DocIdSetBuilder#grow()
                              at org.apache.lucene.facet.FacetsCollector#collect()
                              at org.apache.lucene.search.MultiCollector$MultiLeafCollector#collect()
                              at org.apache.lucene.search.Weight$DefaultBulkScorer#scoreAll()
                              at org.apache.lucene.search.Weight$DefaultBulkScorer#score()
                              at org.apache.lucene.search.BulkScorer#score()
                              at org.apache.lucene.search.IndexSearcher#search()
0.48%         4072M         perf.SearchPerfTest#_main()
                              at perf.SearchPerfTest#main()
0.44%         3723M         org.apache.lucene.search.ExactPhraseMatcher$1#getImpacts()
                              at org.apache.lucene.search.ImpactsDISI#advanceShallow()
                              at org.apache.lucene.search.ImpactsDISI#advanceTarget()
                              at org.apache.lucene.search.ImpactsDISI#advance()
                              at org.apache.lucene.search.ImpactsDISI#nextDoc()
                              at org.apache.lucene.search.Weight$DefaultBulkScorer#scoreAll()
                              at org.apache.lucene.search.Weight$DefaultBulkScorer#score()
                              at org.apache.lucene.search.BulkScorer#score()
                              at org.apache.lucene.search.IndexSearcher#search()
                              at org.apache.lucene.search.IndexSearcher#search()

mikemccand · 2021-01-12T13:56:53Z

And CPU:

PERCENT       CPU SAMPLES   STACK
3.79%         232348        org.apache.lucene.util.packed.DirectMonotonicReader#get()
                              at org.apache.lucene.codecs.lucene80.Lucene80DocValuesProducer$15#binaryValue()
                              at org.apache.lucene.facet.taxonomy.FastTaxonomyFacetCounts#countAll()
                              at org.apache.lucene.facet.taxonomy.FastTaxonomyFacetCounts#<init>()
                              at perf.SearchTask#go()
                              at perf.TaskThreads$TaskThread#run()
3.55%         217460        org.apache.lucene.codecs.lucene80.Lucene80DocValuesProducer$15#binaryValue()
                              at org.apache.lucene.facet.taxonomy.FastTaxonomyFacetCounts#countAll()
                              at org.apache.lucene.facet.taxonomy.FastTaxonomyFacetCounts#<init>()
                              at perf.SearchTask#go()
                              at perf.TaskThreads$TaskThread#run()
2.70%         165557        org.apache.lucene.util.packed.DirectReader$DirectPackedReader12#get()
                              at org.apache.lucene.util.packed.DirectMonotonicReader#get()
                              at org.apache.lucene.codecs.lucene80.Lucene80DocValuesProducer$15#binaryValue()
                              at org.apache.lucene.facet.taxonomy.FastTaxonomyFacetCounts#countAll()
                              at org.apache.lucene.facet.taxonomy.FastTaxonomyFacetCounts#<init>()
                              at perf.SearchTask#go()
                              at perf.TaskThreads$TaskThread#run()
2.13%         130744        org.apache.lucene.facet.taxonomy.FastTaxonomyFacetCounts#countAll()
                              at org.apache.lucene.facet.taxonomy.FastTaxonomyFacetCounts#<init>()
                              at perf.SearchTask#go()
                              at perf.TaskThreads$TaskThread#run()
1.95%         119568        org.apache.lucene.search.grouping.TermGroupSelector#advanceTo()
                              at org.apache.lucene.search.grouping.SecondPassGroupingCollector#collect()
                              at org.apache.lucene.search.Weight$DefaultBulkScorer#scoreAll()
                              at org.apache.lucene.search.Weight$DefaultBulkScorer#score()
                              at org.apache.lucene.search.BulkScorer#score()
                              at org.apache.lucene.search.IndexSearcher#search()
                              at org.apache.lucene.search.IndexSearcher#search()
                              at perf.SearchTask#go()
                              at perf.TaskThreads$TaskThread#run()
1.74%         106828        org.apache.lucene.codecs.lucene84.Lucene84PostingsReader$BlockImpactsPostingsEnum#advance()
                              at org.apache.lucene.search.ConjunctionDISI#doNext()
                              at org.apache.lucene.search.ConjunctionDISI#nextDoc()
                              at org.apache.lucene.search.Weight$DefaultBulkScorer#scoreAll()
                              at org.apache.lucene.search.Weight$DefaultBulkScorer#score()
                              at org.apache.lucene.search.BulkScorer#score()
                              at org.apache.lucene.search.IndexSearcher#search()
                              at org.apache.lucene.search.IndexSearcher#search()
                              at org.apache.lucene.search.IndexSearcher#search()
                              at org.apache.lucene.search.IndexSearcher#searchAfter()
1.54%         94213         java.nio.ByteBuffer#get()
                              at java.nio.DirectByteBuffer#get()
                              at org.apache.lucene.store.ByteBufferGuard#getBytes()
                              at org.apache.lucene.store.ByteBufferIndexInput#readBytes()
                              at org.apache.lucene.codecs.lucene80.Lucene80DocValuesProducer$15#binaryValue()
                              at org.apache.lucene.facet.taxonomy.FastTaxonomyFacetCounts#countAll()
                              at org.apache.lucene.facet.taxonomy.FastTaxonomyFacetCounts#<init>()
                              at perf.SearchTask#go()
                              at perf.TaskThreads$TaskThread#run()
1.03%         63224         org.apache.lucene.facet.sortedset.SortedSetDocValuesFacetCounts#countOneSegment()
                              at org.apache.lucene.facet.sortedset.SortedSetDocValuesFacetCounts#countAll()
                              at org.apache.lucene.facet.sortedset.SortedSetDocValuesFacetCounts#<init>()
                              at org.apache.lucene.facet.sortedset.SortedSetDocValuesFacetCounts#<init>()
                              at perf.SearchTask#go()
                              at perf.TaskThreads$TaskThread#run()
1.02%         62342         org.apache.lucene.queries.intervals.OrderedIntervalsSource$OrderedIntervalIterator#nextInterval()
                              at org.apache.lucene.queries.intervals.IntervalFilter#nextInterval()
                              at org.apache.lucene.queries.intervals.IntervalScorer$1#matches()
                              at org.apache.lucene.search.Weight$DefaultBulkScorer#scoreAll()
                              at org.apache.lucene.search.Weight$DefaultBulkScorer#score()
                              at org.apache.lucene.search.BulkScorer#score()
                              at org.apache.lucene.search.IndexSearcher#search()
                              at org.apache.lucene.search.IndexSearcher#search()
                              at org.apache.lucene.search.IndexSearcher#search()
                              at org.apache.lucene.search.IndexSearcher#searchAfter()
1.02%         62230         org.apache.lucene.search.PhraseScorer$1#matches()
                              at org.apache.lucene.search.Weight$DefaultBulkScorer#scoreAll()
                              at org.apache.lucene.search.Weight$DefaultBulkScorer#score()
                              at org.apache.lucene.search.BulkScorer#score()
                              at org.apache.lucene.search.IndexSearcher#search()
                              at org.apache.lucene.search.IndexSearcher#search()
                              at org.apache.lucene.search.IndexSearcher#search()
                              at org.apache.lucene.search.IndexSearcher#searchAfter()
                              at org.apache.lucene.search.IndexSearcher#search()
                              at perf.SearchTask#go()
1.01%         61871         org.apache.lucene.index.SingletonSortedSetDocValues#nextDoc()
                              at org.apache.lucene.facet.sortedset.SortedSetDocValuesFacetCounts#countOneSegment()
                              at org.apache.lucene.facet.sortedset.SortedSetDocValuesFacetCounts#countAll()
                              at org.apache.lucene.facet.sortedset.SortedSetDocValuesFacetCounts#<init>()
                              at org.apache.lucene.facet.sortedset.SortedSetDocValuesFacetCounts#<init>()
                              at perf.SearchTask#go()
                              at perf.TaskThreads$TaskThread#run()
0.93%         57187         org.apache.lucene.search.ConjunctionDISI#doNext()
                              at org.apache.lucene.search.ConjunctionDISI#nextDoc()
                              at org.apache.lucene.search.Weight$DefaultBulkScorer#scoreAll()
                              at org.apache.lucene.search.Weight$DefaultBulkScorer#score()
                              at org.apache.lucene.search.BulkScorer#score()
                              at org.apache.lucene.search.IndexSearcher#search()
                              at org.apache.lucene.search.IndexSearcher#search()
                              at org.apache.lucene.search.IndexSearcher#search()
                              at org.apache.lucene.search.IndexSearcher#searchAfter()
                              at org.apache.lucene.search.IndexSearcher#search()
0.82%         50023         org.apache.lucene.search.grouping.TermGroupSelector#advanceTo()
                              at org.apache.lucene.search.grouping.FirstPassGroupingCollector#collect()
                              at org.apache.lucene.search.Weight$DefaultBulkScorer#scoreAll()
                              at org.apache.lucene.search.Weight$DefaultBulkScorer#score()
                              at org.apache.lucene.search.BulkScorer#score()
                              at org.apache.lucene.search.IndexSearcher#search()
                              at org.apache.lucene.search.IndexSearcher#search()
                              at perf.SearchTask#go()
                              at perf.TaskThreads$TaskThread#run()
0.63%         38777         org.apache.lucene.search.Weight$DefaultBulkScorer#scoreAll()
                              at org.apache.lucene.search.Weight$DefaultBulkScorer#score()
                              at org.apache.lucene.search.BulkScorer#score()
                              at org.apache.lucene.search.IndexSearcher#search()
                              at org.apache.lucene.search.IndexSearcher#search()
                              at perf.SearchTask#go()
                              at perf.TaskThreads$TaskThread#run()
0.62%         37725         jdk.internal.misc.Unsafe#convEndian()
                              at jdk.internal.misc.Unsafe#getShortUnaligned()
                              at java.nio.DirectByteBuffer#getShort()
                              at java.nio.DirectByteBuffer#getShort()
                              at org.apache.lucene.store.ByteBufferGuard#getShort()
                              at org.apache.lucene.store.ByteBufferIndexInput$SingleBufferImpl#readShort()
                              at org.apache.lucene.util.packed.DirectReader$DirectPackedReader12#get()
                              at org.apache.lucene.util.packed.DirectMonotonicReader#get()
                              at org.apache.lucene.codecs.lucene80.Lucene80DocValuesProducer$15#binaryValue()
                              at org.apache.lucene.facet.taxonomy.FastTaxonomyFacetCounts#countAll()
0.59%         36204         org.apache.lucene.search.TermScorer#score()
                              at org.apache.lucene.search.ScoreCachingWrappingScorer#score()
                              at org.apache.lucene.search.FieldComparator$RelevanceComparator#compareBottom()
                              at org.apache.lucene.search.grouping.FirstPassGroupingCollector#isCompetitive()
                              at org.apache.lucene.search.grouping.FirstPassGroupingCollector#collect()
                              at org.apache.lucene.search.Weight$DefaultBulkScorer#scoreAll()
                              at org.apache.lucene.search.Weight$DefaultBulkScorer#score()
                              at org.apache.lucene.search.BulkScorer#score()
                              at org.apache.lucene.search.IndexSearcher#search()
                              at org.apache.lucene.search.IndexSearcher#search()
0.56%         34042         org.apache.lucene.codecs.lucene84.Lucene84PostingsReader#findFirstGreater()
                              at org.apache.lucene.codecs.lucene84.Lucene84PostingsReader$BlockImpactsPostingsEnum#advance()
                              at org.apache.lucene.search.ConjunctionDISI#doNext()
                              at org.apache.lucene.search.ConjunctionDISI#nextDoc()
                              at org.apache.lucene.search.Weight$DefaultBulkScorer#scoreAll()
                              at org.apache.lucene.search.Weight$DefaultBulkScorer#score()
                              at org.apache.lucene.search.BulkScorer#score()
                              at org.apache.lucene.search.IndexSearcher#search()
                              at org.apache.lucene.search.IndexSearcher#search()
                              at org.apache.lucene.search.IndexSearcher#search()
0.55%         33870         org.apache.lucene.facet.taxonomy.IntTaxonomyFacets#increment()
                              at org.apache.lucene.facet.taxonomy.IntTaxonomyFacets#increment()
                              at org.apache.lucene.facet.taxonomy.FastTaxonomyFacetCounts#countAll()
                              at org.apache.lucene.facet.taxonomy.FastTaxonomyFacetCounts#<init>()
                              at perf.SearchTask#go()
                              at perf.TaskThreads$TaskThread#run()
0.54%         33294         org.apache.lucene.store.ByteBufferGuard#ensureValid()
                              at org.apache.lucene.store.ByteBufferGuard#getShort()
                              at org.apache.lucene.store.ByteBufferIndexInput$SingleBufferImpl#readShort()
                              at org.apache.lucene.util.packed.DirectReader$DirectPackedReader12#get()
                              at org.apache.lucene.util.packed.DirectMonotonicReader#get()
                              at org.apache.lucene.codecs.lucene80.Lucene80DocValuesProducer$15#binaryValue()
                              at org.apache.lucene.facet.taxonomy.FastTaxonomyFacetCounts#countAll()
                              at org.apache.lucene.facet.taxonomy.FastTaxonomyFacetCounts#<init>()
                              at perf.SearchTask#go()
                              at perf.TaskThreads$TaskThread#run()
0.53%         32211         org.apache.lucene.search.ScoreCachingWrappingScorer#score()
                              at org.apache.lucene.search.FieldComparator$RelevanceComparator#compareBottom()
                              at org.apache.lucene.search.grouping.FirstPassGroupingCollector#isCompetitive()
                              at org.apache.lucene.search.grouping.FirstPassGroupingCollector#collect()
                              at org.apache.lucene.search.Weight$DefaultBulkScorer#scoreAll()
                              at org.apache.lucene.search.Weight$DefaultBulkScorer#score()
                              at org.apache.lucene.search.BulkScorer#score()
                              at org.apache.lucene.search.IndexSearcher#search()
                              at org.apache.lucene.search.IndexSearcher#search()
                              at perf.SearchTask#go()
0.50%         30812         org.apache.lucene.codecs.lucene84.Lucene84PostingsReader$EverythingEnum#skipPositions()
                              at org.apache.lucene.codecs.lucene84.Lucene84PostingsReader$EverythingEnum#nextPosition()
                              at org.apache.lucene.queries.intervals.TermIntervalsSource$1#nextInterval()
                              at org.apache.lucene.queries.intervals.OrderedIntervalsSource$OrderedIntervalIterator#nextInterval()
                              at org.apache.lucene.queries.intervals.IntervalFilter#nextInterval()
                              at org.apache.lucene.queries.intervals.IntervalScorer$1#matches()
                              at org.apache.lucene.search.Weight$DefaultBulkScorer#scoreAll()
                              at org.apache.lucene.search.Weight$DefaultBulkScorer#score()
                              at org.apache.lucene.search.BulkScorer#score()
                              at org.apache.lucene.search.IndexSearcher#search()
0.50%         30709         org.apache.lucene.search.grouping.BlockGroupingCollector#collect()
                              at org.apache.lucene.search.Weight$DefaultBulkScorer#scoreAll()
                              at org.apache.lucene.search.Weight$DefaultBulkScorer#score()
                              at org.apache.lucene.search.BulkScorer#score()
                              at org.apache.lucene.search.IndexSearcher#search()
                              at org.apache.lucene.search.IndexSearcher#search()
                              at perf.SearchTask#go()
                              at perf.TaskThreads$TaskThread#run()
0.49%         30235         org.apache.lucene.search.similarities.BM25Similarity$BM25Scorer#score()
                              at org.apache.lucene.search.LeafSimScorer#score()
                              at org.apache.lucene.search.TermScorer#score()
                              at org.apache.lucene.search.ScoreCachingWrappingScorer#score()
                              at org.apache.lucene.search.FieldComparator$RelevanceComparator#compareBottom()
                              at org.apache.lucene.search.grouping.FirstPassGroupingCollector#isCompetitive()
                              at org.apache.lucene.search.grouping.FirstPassGroupingCollector#collect()
                              at org.apache.lucene.search.Weight$DefaultBulkScorer#scoreAll()
                              at org.apache.lucene.search.Weight$DefaultBulkScorer#score()
                              at org.apache.lucene.search.BulkScorer#score()
0.48%         29554         org.apache.lucene.search.SloppyPhraseMatcher#maxFreq()
                              at org.apache.lucene.search.PhraseScorer$1#matches()
                              at org.apache.lucene.search.Weight$DefaultBulkScorer#scoreAll()
                              at org.apache.lucene.search.Weight$DefaultBulkScorer#score()
                              at org.apache.lucene.search.BulkScorer#score()
                              at org.apache.lucene.search.IndexSearcher#search()
                              at org.apache.lucene.search.IndexSearcher#search()
                              at org.apache.lucene.search.IndexSearcher#search()
                              at org.apache.lucene.search.IndexSearcher#searchAfter()
                              at org.apache.lucene.search.IndexSearcher#search()
0.48%         29119         jdk.internal.misc.Unsafe#convEndian()
                              at jdk.internal.misc.Unsafe#getIntUnaligned()
                              at java.nio.DirectByteBuffer#getInt()
                              at java.nio.DirectByteBuffer#getInt()
                              at org.apache.lucene.store.ByteBufferGuard#getInt()
                              at org.apache.lucene.store.ByteBufferIndexInput$SingleBufferImpl#readInt()
                              at org.apache.lucene.util.packed.DirectReader$DirectPackedReader20#get()
                              at org.apache.lucene.codecs.lucene80.Lucene80DocValuesProducer$21#ordValue()
                              at org.apache.lucene.search.grouping.TermGroupSelector#advanceTo()
                              at org.apache.lucene.search.grouping.SecondPassGroupingCollector#collect()
0.44%         26883         org.apache.lucene.util.packed.DirectReader$DirectPackedReader12#get()
                              at org.apache.lucene.codecs.lucene80.Lucene80DocValuesProducer$21#ordValue()
                              at org.apache.lucene.index.SingletonSortedSetDocValues#nextDoc()
                              at org.apache.lucene.facet.sortedset.SortedSetDocValuesFacetCounts#countOneSegment()
                              at org.apache.lucene.facet.sortedset.SortedSetDocValuesFacetCounts#countAll()
                              at org.apache.lucene.facet.sortedset.SortedSetDocValuesFacetCounts#<init>()
                              at org.apache.lucene.facet.sortedset.SortedSetDocValuesFacetCounts#<init>()
                              at perf.SearchTask#go()
                              at perf.TaskThreads$TaskThread#run()
0.44%         26770         java.nio.Buffer#position()
                              at java.nio.ByteBuffer#position()
                              at java.nio.MappedByteBuffer#position()
                              at java.nio.MappedByteBuffer#position()
                              at org.apache.lucene.store.ByteBufferIndexInput$SingleBufferImpl#seek()
                              at org.apache.lucene.codecs.lucene80.Lucene80DocValuesProducer$15#binaryValue()
                              at org.apache.lucene.facet.taxonomy.FastTaxonomyFacetCounts#countAll()
                              at org.apache.lucene.facet.taxonomy.FastTaxonomyFacetCounts#<init>()
                              at perf.SearchTask#go()
                              at perf.TaskThreads$TaskThread#run()
0.43%         26243         org.apache.lucene.codecs.lucene80.Lucene80NormsProducer$3#longValue()
                              at org.apache.lucene.search.LeafSimScorer#getNormValue()
                              at org.apache.lucene.search.LeafSimScorer#score()
                              at org.apache.lucene.search.TermScorer#score()
                              at org.apache.lucene.search.BlockMaxConjunctionScorer#score()
                              at org.apache.lucene.search.TopScoreDocCollector$SimpleTopScoreDocCollector$1#collect()
                              at org.apache.lucene.search.Weight$DefaultBulkScorer#scoreAll()
                              at org.apache.lucene.search.Weight$DefaultBulkScorer#score()
                              at org.apache.lucene.search.BulkScorer#score()
                              at org.apache.lucene.search.IndexSearcher#search()
0.42%         25962         org.apache.lucene.index.SingletonSortedSetDocValues#nextOrd()
                              at org.apache.lucene.facet.sortedset.SortedSetDocValuesFacetCounts#countOneSegment()
                              at org.apache.lucene.facet.sortedset.SortedSetDocValuesFacetCounts#countAll()
                              at org.apache.lucene.facet.sortedset.SortedSetDocValuesFacetCounts#<init>()
                              at org.apache.lucene.facet.sortedset.SortedSetDocValuesFacetCounts#<init>()
                              at perf.SearchTask#go()
                              at perf.TaskThreads$TaskThread#run()
0.41%         25051         org.apache.lucene.search.Weight$DefaultBulkScorer#scoreAll()
                              at org.apache.lucene.search.Weight$DefaultBulkScorer#score()
                              at org.apache.lucene.search.BulkScorer#score()
                              at org.apache.lucene.search.IndexSearcher#search()
                              at org.apache.lucene.search.IndexSearcher#search()
                              at org.apache.lucene.search.IndexSearcher#search()
                              at org.apache.lucene.search.IndexSearcher#searchAfter()
                              at org.apache.lucene.search.IndexSearcher#search()
                              at perf.SearchTask#go()
                              at perf.TaskThreads$TaskThread#run()

… 1 for interactive benchmarks); also print the full command-line executed to aggregate all search JFR results

msokolov · 2021-01-12T15:55:27Z

The VectorDictionary parsing should be a one-time startup cost that is part of the luceneutil test harness - probably it's not worth optimizing? Although we could certainly produce a more compact easy-to-read dictionary representation if we want to.

…

On Tue, Jan 12, 2021 at 8:57 AM Michael McCandless ***@***.***> wrote: And CPU: PERCENT CPU SAMPLES STACK 3.79% 232348 org.apache.lucene.util.packed.DirectMonotonicReader#get() at org.apache.lucene.codecs.lucene80.Lucene80DocValuesProducer$15#binaryValue() at org.apache.lucene.facet.taxonomy.FastTaxonomyFacetCounts#countAll() at org.apache.lucene.facet.taxonomy.FastTaxonomyFacetCounts#<init>() at perf.SearchTask#go() at perf.TaskThreads$TaskThread#run() 3.55% 217460 org.apache.lucene.codecs.lucene80.Lucene80DocValuesProducer$15#binaryValue() at org.apache.lucene.facet.taxonomy.FastTaxonomyFacetCounts#countAll() at org.apache.lucene.facet.taxonomy.FastTaxonomyFacetCounts#<init>() at perf.SearchTask#go() at perf.TaskThreads$TaskThread#run() 2.70% 165557 org.apache.lucene.util.packed.DirectReader$DirectPackedReader12#get() at org.apache.lucene.util.packed.DirectMonotonicReader#get() at org.apache.lucene.codecs.lucene80.Lucene80DocValuesProducer$15#binaryValue() at org.apache.lucene.facet.taxonomy.FastTaxonomyFacetCounts#countAll() at org.apache.lucene.facet.taxonomy.FastTaxonomyFacetCounts#<init>() at perf.SearchTask#go() at perf.TaskThreads$TaskThread#run() 2.13% 130744 org.apache.lucene.facet.taxonomy.FastTaxonomyFacetCounts#countAll() at org.apache.lucene.facet.taxonomy.FastTaxonomyFacetCounts#<init>() at perf.SearchTask#go() at perf.TaskThreads$TaskThread#run() 1.95% 119568 org.apache.lucene.search.grouping.TermGroupSelector#advanceTo() at org.apache.lucene.search.grouping.SecondPassGroupingCollector#collect() at org.apache.lucene.search.Weight$DefaultBulkScorer#scoreAll() at org.apache.lucene.search.Weight$DefaultBulkScorer#score() at org.apache.lucene.search.BulkScorer#score() at org.apache.lucene.search.IndexSearcher#search() at org.apache.lucene.search.IndexSearcher#search() at perf.SearchTask#go() at perf.TaskThreads$TaskThread#run() 1.74% 106828 org.apache.lucene.codecs.lucene84.Lucene84PostingsReader$BlockImpactsPostingsEnum#advance() at org.apache.lucene.search.ConjunctionDISI#doNext() at org.apache.lucene.search.ConjunctionDISI#nextDoc() at org.apache.lucene.search.Weight$DefaultBulkScorer#scoreAll() at org.apache.lucene.search.Weight$DefaultBulkScorer#score() at org.apache.lucene.search.BulkScorer#score() at org.apache.lucene.search.IndexSearcher#search() at org.apache.lucene.search.IndexSearcher#search() at org.apache.lucene.search.IndexSearcher#search() at org.apache.lucene.search.IndexSearcher#searchAfter() 1.54% 94213 java.nio.ByteBuffer#get() at java.nio.DirectByteBuffer#get() at org.apache.lucene.store.ByteBufferGuard#getBytes() at org.apache.lucene.store.ByteBufferIndexInput#readBytes() at org.apache.lucene.codecs.lucene80.Lucene80DocValuesProducer$15#binaryValue() at org.apache.lucene.facet.taxonomy.FastTaxonomyFacetCounts#countAll() at org.apache.lucene.facet.taxonomy.FastTaxonomyFacetCounts#<init>() at perf.SearchTask#go() at perf.TaskThreads$TaskThread#run() 1.03% 63224 org.apache.lucene.facet.sortedset.SortedSetDocValuesFacetCounts#countOneSegment() at org.apache.lucene.facet.sortedset.SortedSetDocValuesFacetCounts#countAll() at org.apache.lucene.facet.sortedset.SortedSetDocValuesFacetCounts#<init>() at org.apache.lucene.facet.sortedset.SortedSetDocValuesFacetCounts#<init>() at perf.SearchTask#go() at perf.TaskThreads$TaskThread#run() 1.02% 62342 org.apache.lucene.queries.intervals.OrderedIntervalsSource$OrderedIntervalIterator#nextInterval() at org.apache.lucene.queries.intervals.IntervalFilter#nextInterval() at org.apache.lucene.queries.intervals.IntervalScorer$1#matches() at org.apache.lucene.search.Weight$DefaultBulkScorer#scoreAll() at org.apache.lucene.search.Weight$DefaultBulkScorer#score() at org.apache.lucene.search.BulkScorer#score() at org.apache.lucene.search.IndexSearcher#search() at org.apache.lucene.search.IndexSearcher#search() at org.apache.lucene.search.IndexSearcher#search() at org.apache.lucene.search.IndexSearcher#searchAfter() 1.02% 62230 org.apache.lucene.search.PhraseScorer$1#matches() at org.apache.lucene.search.Weight$DefaultBulkScorer#scoreAll() at org.apache.lucene.search.Weight$DefaultBulkScorer#score() at org.apache.lucene.search.BulkScorer#score() at org.apache.lucene.search.IndexSearcher#search() at org.apache.lucene.search.IndexSearcher#search() at org.apache.lucene.search.IndexSearcher#search() at org.apache.lucene.search.IndexSearcher#searchAfter() at org.apache.lucene.search.IndexSearcher#search() at perf.SearchTask#go() 1.01% 61871 org.apache.lucene.index.SingletonSortedSetDocValues#nextDoc() at org.apache.lucene.facet.sortedset.SortedSetDocValuesFacetCounts#countOneSegment() at org.apache.lucene.facet.sortedset.SortedSetDocValuesFacetCounts#countAll() at org.apache.lucene.facet.sortedset.SortedSetDocValuesFacetCounts#<init>() at org.apache.lucene.facet.sortedset.SortedSetDocValuesFacetCounts#<init>() at perf.SearchTask#go() at perf.TaskThreads$TaskThread#run() 0.93% 57187 org.apache.lucene.search.ConjunctionDISI#doNext() at org.apache.lucene.search.ConjunctionDISI#nextDoc() at org.apache.lucene.search.Weight$DefaultBulkScorer#scoreAll() at org.apache.lucene.search.Weight$DefaultBulkScorer#score() at org.apache.lucene.search.BulkScorer#score() at org.apache.lucene.search.IndexSearcher#search() at org.apache.lucene.search.IndexSearcher#search() at org.apache.lucene.search.IndexSearcher#search() at org.apache.lucene.search.IndexSearcher#searchAfter() at org.apache.lucene.search.IndexSearcher#search() 0.82% 50023 org.apache.lucene.search.grouping.TermGroupSelector#advanceTo() at org.apache.lucene.search.grouping.FirstPassGroupingCollector#collect() at org.apache.lucene.search.Weight$DefaultBulkScorer#scoreAll() at org.apache.lucene.search.Weight$DefaultBulkScorer#score() at org.apache.lucene.search.BulkScorer#score() at org.apache.lucene.search.IndexSearcher#search() at org.apache.lucene.search.IndexSearcher#search() at perf.SearchTask#go() at perf.TaskThreads$TaskThread#run() 0.63% 38777 org.apache.lucene.search.Weight$DefaultBulkScorer#scoreAll() at org.apache.lucene.search.Weight$DefaultBulkScorer#score() at org.apache.lucene.search.BulkScorer#score() at org.apache.lucene.search.IndexSearcher#search() at org.apache.lucene.search.IndexSearcher#search() at perf.SearchTask#go() at perf.TaskThreads$TaskThread#run() 0.62% 37725 jdk.internal.misc.Unsafe#convEndian() at jdk.internal.misc.Unsafe#getShortUnaligned() at java.nio.DirectByteBuffer#getShort() at java.nio.DirectByteBuffer#getShort() at org.apache.lucene.store.ByteBufferGuard#getShort() at org.apache.lucene.store.ByteBufferIndexInput$SingleBufferImpl#readShort() at org.apache.lucene.util.packed.DirectReader$DirectPackedReader12#get() at org.apache.lucene.util.packed.DirectMonotonicReader#get() at org.apache.lucene.codecs.lucene80.Lucene80DocValuesProducer$15#binaryValue() at org.apache.lucene.facet.taxonomy.FastTaxonomyFacetCounts#countAll() 0.59% 36204 org.apache.lucene.search.TermScorer#score() at org.apache.lucene.search.ScoreCachingWrappingScorer#score() at org.apache.lucene.search.FieldComparator$RelevanceComparator#compareBottom() at org.apache.lucene.search.grouping.FirstPassGroupingCollector#isCompetitive() at org.apache.lucene.search.grouping.FirstPassGroupingCollector#collect() at org.apache.lucene.search.Weight$DefaultBulkScorer#scoreAll() at org.apache.lucene.search.Weight$DefaultBulkScorer#score() at org.apache.lucene.search.BulkScorer#score() at org.apache.lucene.search.IndexSearcher#search() at org.apache.lucene.search.IndexSearcher#search() 0.56% 34042 org.apache.lucene.codecs.lucene84.Lucene84PostingsReader#findFirstGreater() at org.apache.lucene.codecs.lucene84.Lucene84PostingsReader$BlockImpactsPostingsEnum#advance() at org.apache.lucene.search.ConjunctionDISI#doNext() at org.apache.lucene.search.ConjunctionDISI#nextDoc() at org.apache.lucene.search.Weight$DefaultBulkScorer#scoreAll() at org.apache.lucene.search.Weight$DefaultBulkScorer#score() at org.apache.lucene.search.BulkScorer#score() at org.apache.lucene.search.IndexSearcher#search() at org.apache.lucene.search.IndexSearcher#search() at org.apache.lucene.search.IndexSearcher#search() 0.55% 33870 org.apache.lucene.facet.taxonomy.IntTaxonomyFacets#increment() at org.apache.lucene.facet.taxonomy.IntTaxonomyFacets#increment() at org.apache.lucene.facet.taxonomy.FastTaxonomyFacetCounts#countAll() at org.apache.lucene.facet.taxonomy.FastTaxonomyFacetCounts#<init>() at perf.SearchTask#go() at perf.TaskThreads$TaskThread#run() 0.54% 33294 org.apache.lucene.store.ByteBufferGuard#ensureValid() at org.apache.lucene.store.ByteBufferGuard#getShort() at org.apache.lucene.store.ByteBufferIndexInput$SingleBufferImpl#readShort() at org.apache.lucene.util.packed.DirectReader$DirectPackedReader12#get() at org.apache.lucene.util.packed.DirectMonotonicReader#get() at org.apache.lucene.codecs.lucene80.Lucene80DocValuesProducer$15#binaryValue() at org.apache.lucene.facet.taxonomy.FastTaxonomyFacetCounts#countAll() at org.apache.lucene.facet.taxonomy.FastTaxonomyFacetCounts#<init>() at perf.SearchTask#go() at perf.TaskThreads$TaskThread#run() 0.53% 32211 org.apache.lucene.search.ScoreCachingWrappingScorer#score() at org.apache.lucene.search.FieldComparator$RelevanceComparator#compareBottom() at org.apache.lucene.search.grouping.FirstPassGroupingCollector#isCompetitive() at org.apache.lucene.search.grouping.FirstPassGroupingCollector#collect() at org.apache.lucene.search.Weight$DefaultBulkScorer#scoreAll() at org.apache.lucene.search.Weight$DefaultBulkScorer#score() at org.apache.lucene.search.BulkScorer#score() at org.apache.lucene.search.IndexSearcher#search() at org.apache.lucene.search.IndexSearcher#search() at perf.SearchTask#go() 0.50% 30812 org.apache.lucene.codecs.lucene84.Lucene84PostingsReader$EverythingEnum#skipPositions() at org.apache.lucene.codecs.lucene84.Lucene84PostingsReader$EverythingEnum#nextPosition() at org.apache.lucene.queries.intervals.TermIntervalsSource$1#nextInterval() at org.apache.lucene.queries.intervals.OrderedIntervalsSource$OrderedIntervalIterator#nextInterval() at org.apache.lucene.queries.intervals.IntervalFilter#nextInterval() at org.apache.lucene.queries.intervals.IntervalScorer$1#matches() at org.apache.lucene.search.Weight$DefaultBulkScorer#scoreAll() at org.apache.lucene.search.Weight$DefaultBulkScorer#score() at org.apache.lucene.search.BulkScorer#score() at org.apache.lucene.search.IndexSearcher#search() 0.50% 30709 org.apache.lucene.search.grouping.BlockGroupingCollector#collect() at org.apache.lucene.search.Weight$DefaultBulkScorer#scoreAll() at org.apache.lucene.search.Weight$DefaultBulkScorer#score() at org.apache.lucene.search.BulkScorer#score() at org.apache.lucene.search.IndexSearcher#search() at org.apache.lucene.search.IndexSearcher#search() at perf.SearchTask#go() at perf.TaskThreads$TaskThread#run() 0.49% 30235 org.apache.lucene.search.similarities.BM25Similarity$BM25Scorer#score() at org.apache.lucene.search.LeafSimScorer#score() at org.apache.lucene.search.TermScorer#score() at org.apache.lucene.search.ScoreCachingWrappingScorer#score() at org.apache.lucene.search.FieldComparator$RelevanceComparator#compareBottom() at org.apache.lucene.search.grouping.FirstPassGroupingCollector#isCompetitive() at org.apache.lucene.search.grouping.FirstPassGroupingCollector#collect() at org.apache.lucene.search.Weight$DefaultBulkScorer#scoreAll() at org.apache.lucene.search.Weight$DefaultBulkScorer#score() at org.apache.lucene.search.BulkScorer#score() 0.48% 29554 org.apache.lucene.search.SloppyPhraseMatcher#maxFreq() at org.apache.lucene.search.PhraseScorer$1#matches() at org.apache.lucene.search.Weight$DefaultBulkScorer#scoreAll() at org.apache.lucene.search.Weight$DefaultBulkScorer#score() at org.apache.lucene.search.BulkScorer#score() at org.apache.lucene.search.IndexSearcher#search() at org.apache.lucene.search.IndexSearcher#search() at org.apache.lucene.search.IndexSearcher#search() at org.apache.lucene.search.IndexSearcher#searchAfter() at org.apache.lucene.search.IndexSearcher#search() 0.48% 29119 jdk.internal.misc.Unsafe#convEndian() at jdk.internal.misc.Unsafe#getIntUnaligned() at java.nio.DirectByteBuffer#getInt() at java.nio.DirectByteBuffer#getInt() at org.apache.lucene.store.ByteBufferGuard#getInt() at org.apache.lucene.store.ByteBufferIndexInput$SingleBufferImpl#readInt() at org.apache.lucene.util.packed.DirectReader$DirectPackedReader20#get() at org.apache.lucene.codecs.lucene80.Lucene80DocValuesProducer$21#ordValue() at org.apache.lucene.search.grouping.TermGroupSelector#advanceTo() at org.apache.lucene.search.grouping.SecondPassGroupingCollector#collect() 0.44% 26883 org.apache.lucene.util.packed.DirectReader$DirectPackedReader12#get() at org.apache.lucene.codecs.lucene80.Lucene80DocValuesProducer$21#ordValue() at org.apache.lucene.index.SingletonSortedSetDocValues#nextDoc() at org.apache.lucene.facet.sortedset.SortedSetDocValuesFacetCounts#countOneSegment() at org.apache.lucene.facet.sortedset.SortedSetDocValuesFacetCounts#countAll() at org.apache.lucene.facet.sortedset.SortedSetDocValuesFacetCounts#<init>() at org.apache.lucene.facet.sortedset.SortedSetDocValuesFacetCounts#<init>() at perf.SearchTask#go() at perf.TaskThreads$TaskThread#run() 0.44% 26770 java.nio.Buffer#position() at java.nio.ByteBuffer#position() at java.nio.MappedByteBuffer#position() at java.nio.MappedByteBuffer#position() at org.apache.lucene.store.ByteBufferIndexInput$SingleBufferImpl#seek() at org.apache.lucene.codecs.lucene80.Lucene80DocValuesProducer$15#binaryValue() at org.apache.lucene.facet.taxonomy.FastTaxonomyFacetCounts#countAll() at org.apache.lucene.facet.taxonomy.FastTaxonomyFacetCounts#<init>() at perf.SearchTask#go() at perf.TaskThreads$TaskThread#run() 0.43% 26243 org.apache.lucene.codecs.lucene80.Lucene80NormsProducer$3#longValue() at org.apache.lucene.search.LeafSimScorer#getNormValue() at org.apache.lucene.search.LeafSimScorer#score() at org.apache.lucene.search.TermScorer#score() at org.apache.lucene.search.BlockMaxConjunctionScorer#score() at org.apache.lucene.search.TopScoreDocCollector$SimpleTopScoreDocCollector$1#collect() at org.apache.lucene.search.Weight$DefaultBulkScorer#scoreAll() at org.apache.lucene.search.Weight$DefaultBulkScorer#score() at org.apache.lucene.search.BulkScorer#score() at org.apache.lucene.search.IndexSearcher#search() 0.42% 25962 org.apache.lucene.index.SingletonSortedSetDocValues#nextOrd() at org.apache.lucene.facet.sortedset.SortedSetDocValuesFacetCounts#countOneSegment() at org.apache.lucene.facet.sortedset.SortedSetDocValuesFacetCounts#countAll() at org.apache.lucene.facet.sortedset.SortedSetDocValuesFacetCounts#<init>() at org.apache.lucene.facet.sortedset.SortedSetDocValuesFacetCounts#<init>() at perf.SearchTask#go() at perf.TaskThreads$TaskThread#run() 0.41% 25051 org.apache.lucene.search.Weight$DefaultBulkScorer#scoreAll() at org.apache.lucene.search.Weight$DefaultBulkScorer#score() at org.apache.lucene.search.BulkScorer#score() at org.apache.lucene.search.IndexSearcher#search() at org.apache.lucene.search.IndexSearcher#search() at org.apache.lucene.search.IndexSearcher#search() at org.apache.lucene.search.IndexSearcher#searchAfter() at org.apache.lucene.search.IndexSearcher#search() at perf.SearchTask#go() at perf.TaskThreads$TaskThread#run() — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#77 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAHHUQMKUHMEZKUEHDQNOXLSZRILLANCNFSM4PRCRYEA> .

mikemccand · 2021-01-12T19:40:06Z

Good point -- that is a one-time cost, amortized over the life of a service that then handles query traffic.

Can I somehow instruct JFR to NOT profile at startup, and then trigger (from within my Java program, not via external JMC or so) for JFR to start?

rmuir · 2021-01-16T02:32:03Z

Mike, there are various ways to do this. For this case, maybe simply filter it out in the analysis? That way you have precise control. You can look at method / stacktrace / whatever to determine whatever logic you want, e.g. ignoring anything coming from perf.TaskParser#<init>() or whatever is appropriate here to remove ghosts.

Similar stuff is already done in unit tests for the gradle epoll thread: https://github.com/apache/lucene-solr/blob/master/buildSrc/src/main/java/org/apache/lucene/gradle/ProfileResults.java#L101

rmuir · 2021-01-16T02:42:42Z

Another alternative (untested, based on my understanding): If you can start the thing programmatically in the spawned jvms, then don't pass all the options as -XXStartFlightRecording= instead pass them as -XX:FlightRecorderOptions=.

In order to "start it" you then need to issue a diagnostic command. Usually done with jcmd executable, but i found some snippets here that show how to send the diagnostic with mbean: https://github.com/jmiettinen/jfr/blob/master/src/main/java/fi/eonwe/JFR.java

…r results into visible nightly logs

mikemccand · 2021-01-18T14:20:42Z

Last night's benchmarks ran with indexer profiling too:

Profiler for cpu
PROFILE SUMMARY from 471699 events (total: 471699)
  tests.profile.mode=cpu
  tests.profile.count=30
  tests.profile.stacksize=5
  tests.profile.linenumbers=false
PERCENT       CPU SAMPLES   STACK
6.72%         31682         org.apache.lucene.util.ByteBlockPool#setBytesRef()
                              at org.apache.lucene.util.BytesRefHash#equals()
                              at org.apache.lucene.util.BytesRefHash#findHash()
                              at org.apache.lucene.util.BytesRefHash#add()
                              at org.apache.lucene.index.TermsHashPerField#add()
6.63%         31278         org.apache.lucene.util.VectorUtil#dotProduct()
                              at org.apache.lucene.index.VectorValues$SearchStrategy#compare()
                              at org.apache.lucene.util.hnsw.HnswGraph#search()
                              at org.apache.lucene.util.hnsw.HnswGraphBuilder#addGraphNode()
                              at org.apache.lucene.util.hnsw.HnswGraphBuilder#build()
6.19%         29192         org.apache.lucene.analysis.standard.StandardTokenizerImpl#getNextToken()
                              at org.apache.lucene.analysis.standard.StandardTokenizer#incrementToken()
                              at org.apache.lucene.analysis.LowerCaseFilter#incrementToken()
                              at org.apache.lucene.analysis.FilteringTokenFilter#incrementToken()
                              at org.apache.lucene.index.IndexingChain$PerField#invert()
5.22%         24602         org.apache.lucene.index.TermsHashPerField#writeByte()
                              at org.apache.lucene.index.TermsHashPerField#writeVInt()
                              at org.apache.lucene.index.FreqProxTermsWriterPerField#addTerm()
                              at org.apache.lucene.index.TermsHashPerField#positionStreamSlice()
                              at org.apache.lucene.index.TermsHashPerField#add()
4.17%         19684         org.apache.lucene.util.BytesRefHash#findHash()
                              at org.apache.lucene.util.BytesRefHash#add()
                              at org.apache.lucene.index.TermsHashPerField#add()
                              at org.apache.lucene.index.IndexingChain$PerField#invert()
                              at org.apache.lucene.index.IndexingChain#processField()
2.75%         12994         java.io.BufferedOutputStream#write()
                              at org.apache.lucene.store.OutputStreamIndexOutput#writeByte()
                              at org.apache.lucene.store.RateLimitedIndexOutput#writeByte()
                              at org.apache.lucene.store.DataOutput#writeInt()
                              at org.apache.lucene.store.DataOutput#writeLong()
2.70%         12721         java.util.Arrays#binarySearch0()
                              at java.util.Arrays#binarySearch()
                              at org.apache.lucene.codecs.VectorWriter$VectorValuesMerger$MergerRandomAccess#vectorValue()
                              at org.apache.lucene.util.hnsw.HnswGraph#search()
                              at org.apache.lucene.util.hnsw.HnswGraphBuilder#addGraphNode()
2.62%         12344         org.apache.lucene.analysis.standard.StandardAnalyzer#lambda$createComponents$0()
                              at org.apache.lucene.analysis.standard.StandardAnalyzer$$Lambda$229+0x0000000800c25f98.1329879585#accept()
                              at org.apache.lucene.analysis.Analyzer$TokenStreamComponents#setReader()
                              at org.apache.lucene.analysis.Analyzer#tokenStream()
                              at org.apache.lucene.document.Field#tokenStream()
2.42%         11399         sun.nio.ch.FileDispatcherImpl#write0()
                              at sun.nio.ch.FileDispatcherImpl#write()
                              at sun.nio.ch.IOUtil#writeFromNativeBuffer()
                              at sun.nio.ch.IOUtil#write()
                              at sun.nio.ch.FileChannelImpl#write()
2.18%         10286         org.apache.lucene.index.TermsHashPerField#writeByte()
                              at org.apache.lucene.index.TermsHashPerField#writeVInt()
                              at org.apache.lucene.index.FreqProxTermsWriterPerField#writeProx()
                              at org.apache.lucene.index.FreqProxTermsWriterPerField#addTerm()
                              at org.apache.lucene.index.TermsHashPerField#positionStreamSlice()
1.92%         9072          org.apache.lucene.util.LongHeap#push()
                              at org.apache.lucene.util.hnsw.NeighborQueue#add()
                              at org.apache.lucene.util.hnsw.HnswGraph#search()
                              at org.apache.lucene.util.hnsw.HnswGraphBuilder#addGraphNode()
                              at org.apache.lucene.util.hnsw.HnswGraphBuilder#build()
1.92%         9055          org.apache.lucene.index.FreqProxTermsWriterPerField#addTerm()
                              at org.apache.lucene.index.TermsHashPerField#positionStreamSlice()
                              at org.apache.lucene.index.TermsHashPerField#add()
                              at org.apache.lucene.index.IndexingChain$PerField#invert()
                              at org.apache.lucene.index.IndexingChain#processField()
1.65%         7762          java.io.BufferedOutputStream#write()
                              at org.apache.lucene.store.OutputStreamIndexOutput#writeByte()
                              at org.apache.lucene.store.DataOutput#writeInt()
                              at org.apache.lucene.store.DataOutput#writeLong()
                              at org.apache.lucene.codecs.lucene84.ForUtil#encode()
1.57%         7390          org.apache.lucene.index.VectorValues$SearchStrategy#compare()
                              at org.apache.lucene.util.hnsw.HnswGraph#search()
                              at org.apache.lucene.util.hnsw.HnswGraphBuilder#addGraphNode()
                              at org.apache.lucene.util.hnsw.HnswGraphBuilder#build()
                              at org.apache.lucene.codecs.lucene90.Lucene90VectorWriter#writeGraph()
1.56%         7358          org.apache.lucene.util.hnsw.HnswGraph#search()
                              at org.apache.lucene.util.hnsw.HnswGraphBuilder#addGraphNode()
                              at org.apache.lucene.util.hnsw.HnswGraphBuilder#build()
                              at org.apache.lucene.codecs.lucene90.Lucene90VectorWriter#writeGraph()
                              at org.apache.lucene.codecs.lucene90.Lucene90VectorWriter#writeField()

and:

Profiler for heap
PROFILE SUMMARY from 134203 events (total: 493457M)
  tests.profile.mode=heap
  tests.profile.count=30
  tests.profile.stacksize=5
  tests.profile.linenumbers=false
PERCENT       HEAP SAMPLES  STACK
27.11%        133792M       org.apache.lucene.util.fst.BytesStore#writeByte()
                              at org.apache.lucene.util.fst.FST#<init>()
                              at org.apache.lucene.util.fst.FSTCompiler#<init>()
                              at org.apache.lucene.util.fst.FSTCompiler$Builder#build()
                              at org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$PendingBlock#compileIndex()
9.92%         48968M        perf.LineFileDocs#nextDoc()
                              at perf.IndexThreads$IndexThread#run()
7.29%         35963M        java.lang.String#<init>()
                              at java.lang.String#<init>()
                              at perf.LineFileDocs#nextDoc()
                              at perf.IndexThreads$IndexThread#run()
5.76%         28415M        org.apache.lucene.util.SparseFixedBitSet#insertLong()
                              at org.apache.lucene.util.SparseFixedBitSet#set()
                              at org.apache.lucene.util.hnsw.HnswGraph#search()
                              at org.apache.lucene.util.hnsw.HnswGraphBuilder#addGraphNode()
                              at org.apache.lucene.util.hnsw.HnswGraphBuilder#build()
5.16%         25465M        java.lang.StringUTF16#compress()
                              at java.lang.String#<init>()
                              at java.lang.String#<init>()
                              at perf.LineFileDocs#nextDoc()
                              at perf.IndexThreads$IndexThread#run()
5.08%         25073M        perf.LineFileDocs#readDocs()
                              at perf.LineFileDocs$1#run()
4.45%         21942M        org.apache.lucene.codecs.lucene84.Lucene84PostingsWriter#newTermState()
                              at org.apache.lucene.codecs.lucene84.Lucene84PostingsWriter#newTermState()
                              at org.apache.lucene.codecs.PushPostingsWriterBase#writeTerm()
                              at org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter#write()
                              at org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter#write()
4.10%         20237M        org.apache.lucene.index.FreqProxTermsWriterPerField$FreqProxPostingsArray#<init>()
                              at org.apache.lucene.index.FreqProxTermsWriterPerField$FreqProxPostingsArray#newInstance()
                              at org.apache.lucene.index.ParallelPostingsArray#grow()
                              at org.apache.lucene.index.TermsHashPerField$PostingsBytesStartArray#grow()
                              at org.apache.lucene.util.BytesRefHash#add()
2.75%         13555M        org.apache.lucene.util.ArrayUtil#growExact()
                              at org.apache.lucene.util.ArrayUtil#grow()
                              at org.apache.lucene.util.LongHeap#push()
                              at org.apache.lucene.util.hnsw.NeighborQueue#add()
                              at org.apache.lucene.util.hnsw.HnswGraph#search()
2.68%         13211M        org.apache.lucene.index.ParallelPostingsArray#<init>()
                              at org.apache.lucene.index.FreqProxTermsWriterPerField$FreqProxPostingsArray#<init>()
                              at org.apache.lucene.index.FreqProxTermsWriterPerField$FreqProxPostingsArray#newInstance()
                              at org.apache.lucene.index.ParallelPostingsArray#grow()
                              at org.apache.lucene.index.TermsHashPerField$PostingsBytesStartArray#grow()
2.02%         9959M         org.apache.lucene.util.ByteBlockPool$DirectTrackingAllocator#getByteBlock()
                              at org.apache.lucene.util.ByteBlockPool#nextBuffer()
                              at org.apache.lucene.util.ByteBlockPool#allocSlice()
                              at org.apache.lucene.index.TermsHashPerField#writeByte()
                              at org.apache.lucene.index.TermsHashPerField#writeVInt()
1.27%         6283M         org.apache.lucene.util.SparseFixedBitSet#insertBlock()
                              at org.apache.lucene.util.SparseFixedBitSet#set()
                              at org.apache.lucene.util.hnsw.HnswGraph#search()
                              at org.apache.lucene.util.hnsw.HnswGraphBuilder#addGraphNode()
                              at org.apache.lucene.util.hnsw.HnswGraphBuilder#build()
1.15%         5667M         org.apache.lucene.util.BytesRef#<init>()
                              at org.apache.lucene.store.ByteBuffersDataOutput#writeString()
                              at org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter#writeField()
                              at org.apache.lucene.index.StoredFieldsConsumer#writeField()
                              at org.apache.lucene.index.IndexingChain#processField()
1.13%         5567M         org.apache.lucene.codecs.lucene80.Lucene80NormsProducer#getNorms()
                              at org.apache.lucene.codecs.PushPostingsWriterBase#writeTerm()
                              at org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter#write()
                              at org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter#write()
                              at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter#write()
1.02%         5030M         org.apache.lucene.util.SparseFixedBitSet#<init>()
                              at org.apache.lucene.util.hnsw.HnswGraph#search()
                              at org.apache.lucene.util.hnsw.HnswGraphBuilder#addGraphNode()
                              at org.apache.lucene.util.hnsw.HnswGraphBuilder#build()
                              at org.apache.lucene.codecs.lucene90.Lucene90VectorWriter#writeGraph()

It's curious that building FSTs is the top allocator (by sampling)! And, that SparseFixedBitSet.get/insertLong is such a hot spot!

I will fix nightly benchmark to include these profiler outputs in the accessible nightly log.

…ack size and top N

…anular aggregation for cpu/heap

gunnarmorling · 2021-12-20T08:17:51Z

Hi, I just came across this issue, and as you were discussing JFR, I thought the JfrUnit project might be useful here. It lets you run assertions against JFR events, allowing you to identify regressions e.g. in terms of object allocations or I/O triggered by specific operations. Of course, you could also use it to assert custom JFR events (which in turn you could inject via the JMC Agent project, in case there's no suitable events matching your needs).

mikemccand added a commit that referenced this issue Jan 12, 2021

#77: enable Java Flight Recorder (JFR) for interactive search benchma…

de5ee06

…rks and in nightly search benchmarks

mikemccand added a commit that referenced this issue Jan 12, 2021

#77: see depth 12 stacks in nightly benchmarks profiler output (still…

55d98a7

… 1 for interactive benchmarks); also print the full command-line executed to aggregate all search JFR results

mikemccand added a commit that referenced this issue Jan 12, 2021

#77: fix silly syntax errors

8e0f046

mikemccand added a commit that referenced this issue Jan 18, 2021

#77: enable JFR profiling for indexing too; still need to get profile…

62acb20

…r results into visible nightly logs

mikemccand added a commit that referenced this issue Jan 18, 2021

#77: include indexing profiler results in public logs and increase st…

e89a565

…ack size and top N

mikemccand added a commit that referenced this issue Jan 19, 2021

#77: add anchor links to navigate the profiling results

3d7ec6c

rmuir mentioned this issue Feb 4, 2021

Confirm new JFR profiling is adding minimal overhead #105

Open

mikemccand added a commit that referenced this issue Feb 22, 2021

#77: run JFR aggregation at multiple stack depths to get more/less gr…

070a6f8

…anular aggregation for cpu/heap

Low-overhead profiling #77

Low-overhead profiling #77

rmuir commented Jul 31, 2020

rmuir commented Nov 29, 2020

mikemccand commented Jan 5, 2021

mikemccand commented Jan 5, 2021

mikemccand commented Jan 10, 2021

mikemccand commented Jan 12, 2021

mikemccand commented Jan 12, 2021

mikemccand commented Jan 12, 2021

msokolov commented Jan 12, 2021 via email

mikemccand commented Jan 12, 2021

rmuir commented Jan 16, 2021

rmuir commented Jan 16, 2021

mikemccand commented Jan 18, 2021

gunnarmorling commented Dec 20, 2021

Low-overhead profiling #77

Low-overhead profiling #77

Comments

rmuir commented Jul 31, 2020

rmuir commented Nov 29, 2020

mikemccand commented Jan 5, 2021

mikemccand commented Jan 5, 2021

mikemccand commented Jan 10, 2021

mikemccand commented Jan 12, 2021

mikemccand commented Jan 12, 2021

mikemccand commented Jan 12, 2021

msokolov commented Jan 12, 2021 via email

mikemccand commented Jan 12, 2021

rmuir commented Jan 16, 2021

rmuir commented Jan 16, 2021

mikemccand commented Jan 18, 2021

gunnarmorling commented Dec 20, 2021