Skip to content

Support intra-segment search in the query profiler with per-slice and per-partition breakdowns#16417

Open
prudhvigodithi wants to merge 3 commits into
apache:mainfrom
prudhvigodithi:profile
Open

Support intra-segment search in the query profiler with per-slice and per-partition breakdowns#16417
prudhvigodithi wants to merge 3 commits into
apache:mainfrom
prudhvigodithi:profile

Conversation

@prudhvigodithi

Copy link
Copy Markdown
Contributor

Description

Building on intra-segment search concurrency (#13542) and concurrent-search profiling (#14413), this PR attributes query-profiler timings per slice and per partition, resolving the slice-identification problem left open in #14413's review. #14413 keys leaf breakdowns by Thread, so when one thread runs several slices sequentially their timings merge into one bucket and for intra-segment search it cannot attribute timing to the doc-id-range partitions a segment is split into.

Change

  • Reliably identifies slices via a counter assigned at the once-per-slice search(LeafReaderContextPartition[]) seam, independent of thread.
  • Keys leaf timings per partition (sliceId, segmentOrd, minDocId, maxDocId), set at searchLeaf(...); both overrides delegate to super (no duplicated search logic).
  • Single code path — no concurrent/non-concurrent class split; non-concurrent search is the single-slice degenerate case (one slice covering whole-segment partitions).
  • Per-slice threadId/threadName/startTime show how the search parallelized, Thread object replaced by serializable id+name

Sample output

Verified locally by adding a temporary toString() (not included in this PR) and serializing getProfileResult() for a BooleanQuery(+foo:bar foo:baz) over a single segment split into two partitions [0,100) / [100,200) on a 2-thread executor.

[
   {
     "type": "BooleanQuery",
     "description": "+foo:bar foo:baz",
     "breakdown": { "create_weight": 1634167, "create_weight_count": 1 },
     "slices": [
       {
         "slice_id": 0,
         "thread_id": 22,
         "thread_name": "search-worker-caller",
         "partitions": [
           { "segment_ord": 0, "doc_range": [0, 100],
             "breakdown": { "build_scorer": 5185916, "next_doc_count": 101, "score_count": 100, "advance_count": 0 } }
         ]
       },
       {
         "slice_id": 1,
         "thread_id": 23,
         "thread_name": "search-worker-pool-3-thread-1",
         "partitions": [
           { "segment_ord": 0, "doc_range": [100, 200],
             "breakdown": { "build_scorer": 5183792, "next_doc_count": 100, "score_count": 100, "advance_count": 1 } }
         ]
       }
     ],
     "children": [
       {
         "type": "TermQuery", "description": "foo:bar",
         "breakdown": { "create_weight": 903875, "create_weight_count": 1 },
         "slices": [
           { "slice_id": 0, "thread_id": 22, "thread_name": "search-worker-caller",
             "partitions": [ { "segment_ord": 0, "doc_range": [0, 100],
               "breakdown": { "build_scorer": 1026292, "next_doc_count": 101, "score_count": 100 } } ] },
           { "slice_id": 1, "thread_id": 23, "thread_name": "search-worker-pool-3-thread-1",
             "partitions": [ { "segment_ord": 0, "doc_range": [100, 200],
               "breakdown": { "build_scorer": 1008001, "next_doc_count": 100, "score_count": 100 } } ] }
         ],
         "children": []
       },
       {
         "type": "TermQuery", "description": "foo:baz",
         "breakdown": { "create_weight": 52459, "create_weight_count": 1 },
         "slices": [
           { "slice_id": 0, "thread_id": 22, "thread_name": "search-worker-caller",
             "partitions": [ { "segment_ord": 0, "doc_range": [0, 100],
               "breakdown": { "build_scorer": 959, "next_doc_count": 0, "score_count": 0 } } ] },
           { "slice_id": 1, "thread_id": 23, "thread_name": "search-worker-pool-3-thread-1",
             "partitions": [ { "segment_ord": 0, "doc_range": [100, 200],
               "breakdown": { "build_scorer": 708, "next_doc_count": 0, "score_count": 0 } } ] }
         ],
         "children": []
       }
     ]
   }
 ]

Signed-off-by: Prudhvi Godithi <pgodithi@amazon.com>
Signed-off-by: Prudhvi Godithi <pgodithi@amazon.com>
Signed-off-by: Prudhvi Godithi <pgodithi@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant