LUCENE-10061: Implements dynamic pruning support for CombinedFieldsQuery #418
Commits on Oct 29, 2021
-
LUCENE-10061: Implements basic dynamic pruning support for CombinedFi…
…eldsQuery // HighMed: from office # freq=3224339 freq=225338 // top scores time usage 425 milliseconds // complete time usage 401 milliseconds // HighHigh: but publisher # freq=1456553 freq=1289029 // top scores time usage 469 milliseconds // complete time usage 322 milliseconds // HighLow: with fung # freq=3709421 freq=1344 // top scores time usage 241 milliseconds // complete time usage 428 milliseconds // HighLow: date insult # freq=2020626 freq=4424 // top scores time usage 171 milliseconds // complete time usage 225 milliseconds
Commits on Nov 4, 2021
-
replace cartesian product calculation with simple upper-bound
Perf tests result Run 1: TaskQPS baseline StdDevQPS my_modified_version StdDev Pct diff p-value CFQHighHigh 3.53 (3.3%) 2.92 (5.3%) -17.2% ( -25% - -8%) 0.000 PKLookup 108.13 (7.7%) 119.85 (8.2%) 10.8% ( -4% - 28%) 0.000 CFQHighLow 14.88 (3.9%) 16.69 (12.5%) 12.2% ( -3% - 29%) 0.000 CFQHighMed 21.11 (4.1%) 25.87 (11.8%) 22.6% ( 6% - 40%) 0.000 Run 2: TaskQPS baseline StdDevQPS my_modified_version StdDev Pct diff p-value CFQHighHigh 6.64 (3.1%) 5.63 (10.2%) -15.2% ( -27% - -1%) 0.000 CFQHighLow 8.35 (2.8%) 8.05 (15.0%) -3.6% ( -20% - 14%) 0.297 CFQHighMed 24.51 (5.3%) 24.90 (19.9%) 1.6% ( -22% - 28%) 0.733 PKLookup 110.06 (10.0%) 128.54 (7.9%) 16.8% ( -1% - 38%) 0.000 Run 3: TaskQPS baseline StdDevQPS my_modified_version StdDev Pct diff p-value CFQHighMed 13.01 (2.9%) 9.82 (7.8%) -24.5% ( -34% - -14%) 0.000 PKLookup 107.85 (8.1%) 111.79 (11.2%) 3.7% ( -14% - 24%) 0.236 CFQHighHigh 4.83 (2.6%) 5.06 (8.6%) 4.7% ( -6% - 16%) 0.018 CFQHighLow 14.95 (3.0%) 18.31 (19.0%) 22.5% ( 0% - 45%) 0.000 Run 4: TaskQPS baseline StdDevQPS my_modified_version StdDev Pct diff p-value CFQHighMed 11.11 (2.9%) 6.69 (4.1%) -39.7% ( -45% - -33%) 0.000 CFQHighLow 27.55 (3.8%) 25.46 (11.0%) -7.6% ( -21% - 7%) 0.003 CFQHighHigh 5.25 (3.2%) 4.96 (6.1%) -5.7% ( -14% - 3%) 0.000 PKLookup 107.61 (6.7%) 121.19 (4.6%) 12.6% ( 1% - 25%) 0.000
Commits on Nov 5, 2021
-
Bug fix and optimize per CPU profiler
Run 1 TaskQPS baseline StdDevQPS my_modified_version StdDev Pct diff p-value CFQHighHighHigh 4.64 (6.5%) 3.30 (4.7%) -29.0% ( -37% - -19%) 0.000 CFQHighHigh 11.09 (6.0%) 9.61 (6.0%) -13.3% ( -23% - -1%) 0.000 PKLookup 103.38 (4.4%) 108.04 (4.3%) 4.5% ( -4% - 13%) 0.001 CFQHighMedLow 10.58 (6.1%) 12.30 (8.7%) 16.2% ( 1% - 33%) 0.000 CFQHighMed 10.70 (7.4%) 15.51 (11.2%) 44.9% ( 24% - 68%) 0.000 CFQHighLowLow 8.18 (8.2%) 12.87 (11.6%) 57.3% ( 34% - 84%) 0.000 CFQHighLow 14.57 (7.5%) 30.81 (15.1%) 111.4% ( 82% - 144%) 0.000 Run 2 TaskQPS baseline StdDevQPS my_modified_version StdDev Pct diff p-value CFQHighHighHigh 5.33 (5.7%) 4.02 (7.7%) -24.4% ( -35% - -11%) 0.000 CFQHighLowLow 17.14 (6.2%) 13.06 (5.4%) -23.8% ( -33% - -13%) 0.000 CFQHighMed 17.37 (5.8%) 14.38 (7.7%) -17.2% ( -29% - -3%) 0.000 PKLookup 103.57 (5.5%) 108.84 (5.9%) 5.1% ( -6% - 17%) 0.005 CFQHighMedLow 11.25 (7.2%) 12.70 (9.0%) 12.9% ( -3% - 31%) 0.000 CFQHighHigh 5.00 (6.2%) 7.54 (12.1%) 51.0% ( 30% - 73%) 0.000 CFQHighLow 21.60 (5.2%) 34.57 (14.1%) 60.0% ( 38% - 83%) 0.000 Run 3 TaskQPS baseline StdDevQPS my_modified_version StdDev Pct diff p-value CFQHighHighHigh 5.40 (6.9%) 4.06 (5.1%) -24.8% ( -34% - -13%) 0.000 CFQHighMedLow 7.64 (7.4%) 5.79 (6.3%) -24.2% ( -35% - -11%) 0.000 CFQHighHigh 11.11 (7.0%) 9.60 (5.9%) -13.6% ( -24% - 0%) 0.000 CFQHighLowLow 21.21 (7.6%) 21.22 (6.6%) 0.0% ( -13% - 15%) 0.993 PKLookup 103.15 (5.9%) 107.60 (6.9%) 4.3% ( -8% - 18%) 0.034 CFQHighLow 21.85 (8.1%) 34.18 (13.5%) 56.4% ( 32% - 84%) 0.000 CFQHighMed 12.07 (8.4%) 19.98 (16.7%) 65.5% ( 37% - 98%) 0.000 Run 4 TaskQPS baseline StdDevQPS my_modified_version StdDev Pct diff p-value CFQHighHigh 8.50 (5.8%) 6.85 (5.2%) -19.5% ( -28% - -8%) 0.000 CFQHighMedLow 10.89 (5.7%) 8.96 (5.4%) -17.8% ( -27% - -7%) 0.000 CFQHighMed 8.41 (5.8%) 7.74 (5.6%) -7.9% ( -18% - 3%) 0.000 CFQHighHighHigh 3.45 (6.7%) 3.38 (5.3%) -2.0% ( -13% - 10%) 0.287 CFQHighLowLow 7.82 (6.4%) 8.20 (7.5%) 4.8% ( -8% - 20%) 0.030 PKLookup 103.50 (5.0%) 110.69 (5.4%) 6.9% ( -3% - 18%) 0.000 CFQHighLow 11.46 (6.0%) 13.16 (6.7%) 14.8% ( 1% - 29%) 0.000 -
Commits on Nov 7, 2021
-
Run 1 TaskQPS baseline StdDevQPS my_modified_version StdDev Pct diff p-value CFQHighMed 16.99 (4.9%) 16.84 (14.9%) -0.9% ( -19% - 19%) 0.790 PKLookup 98.49 (8.7%) 121.99 (9.1%) 23.9% ( 5% - 45%) 0.000 CFQHighHigh 5.82 (4.7%) 7.55 (10.0%) 29.8% ( 14% - 46%) 0.000 CFQHighLow 17.59 (5.2%) 25.01 (17.3%) 42.2% ( 18% - 68%) 0.000 Run 2 TaskQPS baseline StdDevQPS my_modified_version StdDev Pct diff p-value CFQHighMed 18.28 (6.1%) 15.63 (15.6%) -14.5% ( -34% - 7%) 0.000 PKLookup 101.26 (10.0%) 105.39 (12.0%) 4.1% ( -16% - 28%) 0.243 CFQHighHigh 9.96 (5.6%) 10.47 (13.0%) 5.2% ( -12% - 25%) 0.101 CFQHighLow 9.71 (5.8%) 15.71 (34.0%) 61.8% ( 20% - 107%) 0.000
Commits on Nov 19, 2021
Commits on Nov 22, 2021
Commits on Nov 29, 2021
-
Results from python3 src/python/localrun.py -source combinedFieldsUnevenlyWeightedBig: Run 1: TaskQPS baseline StdDevQPS my_modified_version StdDev Pct diff p-value CFQHighHigh 2.81 (10.9%) 2.40 (10.3%) -14.6% ( -32% - 7%) 0.000 CFQHighMed 3.27 (11.8%) 2.91 (11.0%) -11.2% ( -30% - 13%) 0.002 CFQHighLow 16.09 (14.6%) 14.58 (11.9%) -9.4% ( -31% - 20%) 0.025 PKLookup 92.08 (11.6%) 96.23 (13.3%) 4.5% ( -18% - 33%) 0.255 Run 2: TaskQPS baseline StdDevQPS my_modified_version StdDev Pct diff p-value CFQHighLow 15.76 (12.0%) 10.19 (12.9%) -35.3% ( -53% - -11%) 0.000 CFQHighMed 12.01 (13.3%) 8.79 (13.9%) -26.9% ( -47% - 0%) 0.000 CFQHighHigh 5.59 (10.8%) 4.40 (8.4%) -21.2% ( -36% - -2%) 0.000 PKLookup 93.36 (21.9%) 94.29 (22.6%) 1.0% ( -35% - 58%) 0.888
Commits on Dec 1, 2021
-
Address feedback - avoid repeated computation for max weighted field,…
… replace doc freq collection with ImpactsEnum#cost
Commits on Dec 2, 2021
-
Refactoring - use ImpactsMergingUtils in CombinedFieldsQuery
TaskQPS baseline StdDevQPS my_modified_version StdDev Pct diff p-value CFQHighLow 21.45 (13.8%) 15.65 (18.0%) -27.1% ( -51% - 5%) 0.000 CFQHighMed 5.92 (4.2%) 4.52 (12.2%) -23.7% ( -38% - -7%) 0.000 CFQHighHigh 5.52 (5.0%) 4.45 (12.1%) -19.4% ( -34% - -2%) 0.000 PKLookup 110.45 (11.1%) 108.08 (15.7%) -2.1% ( -26% - 27%) 0.617 = Candidate CPU JFR PROFILE SUMMARY from 86268 events (total: 86268) tests.profile.mode=cpu tests.profile.count=30 tests.profile.stacksize=1 tests.profile.linenumbers=false PERCENT CPU SAMPLES STACK 15.48% 13355 org.apache.lucene.sandbox.search.MultiNormsLeafSimScorer$MultiFieldNormValues#advanceExact() 7.37% 6359 org.apache.lucene.search.DisjunctionDISIApproximation#advance() 7.35% 6345 org.apache.lucene.search.similarities.BM25Similarity$BM25Scorer#score() 6.38% 5508 org.apache.lucene.search.DisiPriorityQueue#downHeap() 4.18% 3605 org.apache.lucene.sandbox.search.CombinedFieldQuery$CombinedFieldScorer#freq() 4.10% 3538 org.apache.lucene.search.TopScoreDocCollector$SimpleTopScoreDocCollector$1#collect() 4.01% 3456 org.apache.lucene.search.DisiPriorityQueue#topList() 3.67% 3169 org.apache.lucene.sandbox.search.CombinedFieldQuery$WeightedDisiWrapper#freq() 3.36% 2899 org.apache.lucene.search.DisiPriorityQueue#top() 3.08% 2660 java.lang.Math#round() 2.40% 2067 org.apache.lucene.sandbox.search.MultiNormsLeafSimScorer#getNormValue() 2.38% 2057 org.apache.lucene.sandbox.search.CombinedFieldQuery$CombinedFieldScorer#score() 1.94% 1675 org.apache.lucene.util.SmallFloat#longToInt4() 1.87% 1617 org.apache.lucene.codecs.lucene90.Lucene90NormsProducer$3#longValue() 1.86% 1605 org.apache.lucene.store.ByteBufferGuard#ensureValid() 1.80% 1557 org.apache.lucene.codecs.lucene90.Lucene90PostingsReader#findFirstGreater() 1.50% 1293 org.apache.lucene.codecs.lucene90.Lucene90PostingsReader$BlockImpactsDocsEnum#advance() 1.39% 1200 org.apache.lucene.search.ImpactsDISI#docID() 1.33% 1145 org.apache.lucene.search.DisiPriorityQueue#updateTop() 1.24% 1067 jdk.internal.misc.Unsafe#getByte() 1.18% 1019 org.apache.lucene.store.ByteBufferGuard#getByte() 0.92% 792 org.apache.lucene.util.SmallFloat#intToByte4() 0.76% 659 java.lang.Math#toIntExact() 0.68% 584 org.apache.lucene.codecs.lucene90.Lucene90PostingsReader$BlockImpactsDocsEnum#freq() 0.67% 579 org.apache.lucene.search.ImpactsDISI#nextDoc() 0.61% 530 org.apache.lucene.search.ImpactsDISI#advanceTarget() 0.58% 500 org.apache.lucene.search.Weight$DefaultBulkScorer#scoreAll() 0.57% 492 org.apache.lucene.search.DisiPriorityQueue#prepend() 0.40% 345 org.apache.lucene.codecs.lucene90.PForUtil#innerPrefixSum32() 0.39% 333 org.apache.lucene.codecs.lucene90.ForUtil#expand8() = Baseline CPU JFR PROFILE SUMMARY from 76683 events (total: 76683) tests.profile.mode=cpu tests.profile.count=30 tests.profile.stacksize=1 tests.profile.linenumbers=false PERCENT CPU SAMPLES STACK 16.26% 12468 org.apache.lucene.sandbox.search.MultiNormsLeafSimScorer$MultiFieldNormValues#advanceExact() 8.46% 6490 org.apache.lucene.search.similarities.BM25Similarity$BM25Scorer#score() 8.24% 6315 org.apache.lucene.search.DisjunctionDISIApproximation#nextDoc() 6.13% 4698 org.apache.lucene.search.DisiPriorityQueue#downHeap() 4.79% 3676 org.apache.lucene.search.DisiPriorityQueue#topList() 4.45% 3412 org.apache.lucene.search.TopScoreDocCollector$SimpleTopScoreDocCollector$1#collect() 4.32% 3314 org.apache.lucene.sandbox.search.CombinedFieldQuery$CombinedFieldScorer#freq() 3.25% 2492 java.lang.Math#round() 3.01% 2309 org.apache.lucene.search.DisiPriorityQueue#top() 2.75% 2107 org.apache.lucene.sandbox.search.CombinedFieldQuery$CombinedFieldScorer#score() 2.62% 2010 org.apache.lucene.sandbox.search.CombinedFieldQuery$WeightedDisiWrapper#freq() 2.29% 1753 org.apache.lucene.search.Weight$DefaultBulkScorer#scoreAll() 2.12% 1623 org.apache.lucene.sandbox.search.MultiNormsLeafSimScorer#getNormValue() 2.05% 1569 org.apache.lucene.util.SmallFloat#longToInt4() 1.93% 1481 org.apache.lucene.store.ByteBufferGuard#ensureValid() 1.77% 1358 org.apache.lucene.codecs.lucene90.Lucene90NormsProducer$3#longValue() 1.45% 1110 org.apache.lucene.search.DisiPriorityQueue#updateTop() 1.25% 958 org.apache.lucene.codecs.lucene90.Lucene90PostingsReader$BlockDocsEnum#nextDoc() 1.19% 914 org.apache.lucene.store.ByteBufferGuard#getByte() 1.17% 898 java.lang.Math#toIntExact() 1.06% 815 org.apache.lucene.util.SmallFloat#intToByte4() 1.02% 782 org.apache.lucene.codecs.lucene90.Lucene90PostingsReader$BlockDocsEnum#freq() 0.86% 656 jdk.internal.misc.Unsafe#getByte() 0.68% 518 org.apache.lucene.codecs.lucene90.PForUtil#decodeAndPrefixSum() 0.64% 492 org.apache.lucene.search.DisiPriorityQueue#prepend() 0.49% 375 org.apache.lucene.codecs.lucene90.PForUtil#innerPrefixSum32() 0.42% 322 org.apache.lucene.codecs.lucene90.ForUtil#expand8() 0.31% 239 java.util.zip.Inflater#inflateBytesBytes() 0.30% 228 org.apache.lucene.codecs.lucene90.blocktree.SegmentTermsEnum#seekExact() 0.29% 224 org.apache.lucene.codecs.lucene90.PForUtil#expand32() -
Use getDocIdUpTo to determine lead impacts
TaskQPS baseline StdDevQPS my_modified_version StdDev Pct diff p-value CFQHighHigh 3.68 (6.5%) 2.66 (7.3%) -27.8% ( -39% - -14%) 0.000 CFQHighLow 14.19 (7.9%) 11.31 (8.0%) -20.3% ( -33% - -4%) 0.000 CFQHighMed 12.92 (9.5%) 10.41 (9.9%) -19.4% ( -35% - 0%) 0.000 PKLookup 104.92 (6.7%) 104.49 (9.0%) -0.4% ( -15% - 16%) 0.869 = Candidate CPU JFR PROFILE SUMMARY from 87201 events (total: 87201) tests.profile.mode=cpu tests.profile.count=30 tests.profile.stacksize=1 tests.profile.linenumbers=false PERCENT CPU SAMPLES STACK 15.13% 13191 org.apache.lucene.sandbox.search.MultiNormsLeafSimScorer$MultiFieldNormValues#advanceExact() 7.42% 6466 org.apache.lucene.search.similarities.BM25Similarity$BM25Scorer#score() 7.29% 6354 org.apache.lucene.search.DisjunctionDISIApproximation#advance() 6.79% 5921 org.apache.lucene.search.DisiPriorityQueue#downHeap() 4.34% 3785 org.apache.lucene.search.DisiPriorityQueue#topList() 3.97% 3465 org.apache.lucene.search.TopScoreDocCollector$SimpleTopScoreDocCollector$1#collect() 3.85% 3360 org.apache.lucene.sandbox.search.CombinedFieldQuery$WeightedDisiWrapper#freq() 3.18% 2777 java.lang.Math#round() 3.01% 2624 org.apache.lucene.sandbox.search.CombinedFieldQuery$CombinedFieldScorer#freq() 3.01% 2624 org.apache.lucene.search.DisiPriorityQueue#top() 2.33% 2036 org.apache.lucene.sandbox.search.CombinedFieldQuery$CombinedFieldScorer#score() 2.20% 1917 org.apache.lucene.sandbox.search.MultiNormsLeafSimScorer#getNormValue() 2.07% 1809 org.apache.lucene.util.SmallFloat#longToInt4() 1.81% 1580 org.apache.lucene.codecs.lucene90.Lucene90NormsProducer$3#longValue() 1.78% 1555 org.apache.lucene.codecs.lucene90.Lucene90PostingsReader$BlockImpactsDocsEnum#advance() 1.77% 1545 org.apache.lucene.store.ByteBufferGuard#ensureValid() 1.51% 1316 org.apache.lucene.codecs.lucene90.Lucene90PostingsReader#findFirstGreater() 1.39% 1210 org.apache.lucene.search.DisiPriorityQueue#updateTop() 1.29% 1124 org.apache.lucene.search.ImpactsDISI#docID() 1.22% 1063 jdk.internal.misc.Unsafe#getByte() 1.19% 1037 org.apache.lucene.store.ByteBufferGuard#getByte() 1.08% 938 org.apache.lucene.search.DisiPriorityQueue#prepend() 0.97% 849 org.apache.lucene.search.Weight$DefaultBulkScorer#scoreAll() 0.86% 750 org.apache.lucene.codecs.MultiLevelSkipListReader#skipTo() 0.82% 717 org.apache.lucene.util.SmallFloat#intToByte4() 0.78% 683 java.lang.Math#toIntExact() 0.62% 540 org.apache.lucene.codecs.lucene90.PForUtil#decode() 0.61% 529 org.apache.lucene.codecs.lucene90.Lucene90PostingsReader$BlockImpactsDocsEnum#freq() 0.51% 441 org.apache.lucene.search.ImpactsDISI#nextDoc() 0.50% 435 org.apache.lucene.search.ImpactsDISI#advanceTarget() = Baseline CPU JFR PROFILE SUMMARY from 81578 events (total: 81578) tests.profile.mode=cpu tests.profile.count=30 tests.profile.stacksize=1 tests.profile.linenumbers=false PERCENT CPU SAMPLES STACK 16.38% 13363 org.apache.lucene.sandbox.search.MultiNormsLeafSimScorer$MultiFieldNormValues#advanceExact() 8.67% 7073 org.apache.lucene.search.similarities.BM25Similarity$BM25Scorer#score() 8.05% 6569 org.apache.lucene.search.DisjunctionDISIApproximation#nextDoc() 6.50% 5305 org.apache.lucene.search.DisiPriorityQueue#downHeap() 5.17% 4220 org.apache.lucene.sandbox.search.CombinedFieldQuery$CombinedFieldScorer#freq() 4.40% 3587 org.apache.lucene.search.TopScoreDocCollector$SimpleTopScoreDocCollector$1#collect() 4.30% 3505 org.apache.lucene.search.DisiPriorityQueue#topList() 3.37% 2751 java.lang.Math#round() 2.77% 2263 org.apache.lucene.search.DisiPriorityQueue#top() 2.64% 2156 org.apache.lucene.sandbox.search.CombinedFieldQuery$WeightedDisiWrapper#freq() 2.62% 2135 org.apache.lucene.sandbox.search.CombinedFieldQuery$CombinedFieldScorer#score() 2.12% 1729 org.apache.lucene.sandbox.search.MultiNormsLeafSimScorer#getNormValue() 2.11% 1723 org.apache.lucene.util.SmallFloat#longToInt4() 2.06% 1678 org.apache.lucene.search.Weight$DefaultBulkScorer#scoreAll() 2.01% 1640 org.apache.lucene.store.ByteBufferGuard#ensureValid() 1.77% 1442 org.apache.lucene.codecs.lucene90.Lucene90NormsProducer$3#longValue() 1.62% 1325 org.apache.lucene.search.DisiPriorityQueue#updateTop() 1.26% 1028 org.apache.lucene.codecs.lucene90.Lucene90PostingsReader$BlockDocsEnum#nextDoc() 1.10% 899 java.lang.Math#toIntExact() 1.10% 898 org.apache.lucene.util.SmallFloat#intToByte4() 1.09% 888 org.apache.lucene.codecs.lucene90.Lucene90PostingsReader$BlockDocsEnum#freq() 1.06% 864 org.apache.lucene.store.ByteBufferGuard#getByte() 0.80% 652 jdk.internal.misc.Unsafe#getByte() 0.74% 605 org.apache.lucene.codecs.lucene90.PForUtil#decode() 0.68% 552 org.apache.lucene.search.DisiPriorityQueue#prepend() 0.60% 492 org.apache.lucene.codecs.lucene90.PForUtil#decodeAndPrefixSum() 0.55% 451 java.io.RandomAccessFile#readBytes() 0.38% 314 org.apache.lucene.codecs.lucene90.PForUtil#innerPrefixSum32() 0.38% 307 org.apache.lucene.codecs.lucene90.ForUtil#expand8() 0.32% 260 org.apache.lucene.codecs.lucene90.PForUtil#expand32()
Commits on Dec 8, 2021
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.