[VL] Add sort merge join metrics #3920
Conversation
|
Thanks for opening a pull request! Could you open an issue for this pull request on Github Issues? https://github.com/oap-project/gluten/issues Then could you also rename commit message and pull request title in the following format? See also: |
|
Run Gluten Clickhouse CI |
| "postProjectionOutputVectors" -> SQLMetrics.createMetric( | ||
| sparkContext, | ||
| "number of postProjection output vectors"), | ||
| "finalOutputRows" -> SQLMetrics.createMetric(sparkContext, "number of final output rows"), |
There was a problem hiding this comment.
it's nerver used before, and replaced to numOutputRows in this pr
|
Run Gluten Clickhouse CI |
There was a problem hiding this comment.
@ulysses-you Thanks for your great work. Just two small questions.
| "bufferPreProjectionCpuCount" -> SQLMetrics.createMetric( | ||
| sparkContext, | ||
| "buffer preProject cpu wall time count"), | ||
| "bufferPreProjectionWallNanos" -> SQLMetrics.createNanoTimingMetric( |
There was a problem hiding this comment.
@ulysses-you What are the differences between stream and buffer? How can they be mapped to the left and right in a sort merge join?
There was a problem hiding this comment.
Merge join only supports inner and left outer, so the steam side is always left child and the buffer side is always right child. It's used to show the metrics pre-project for stream/buffer side.
| super.afterAll() | ||
| } | ||
|
|
||
| test("test sort merge join metrics") { |
There was a problem hiding this comment.
@ulysses-you Do we need to add the test with post project? And also add the hash join test for both with pre project and post project?
There was a problem hiding this comment.
Join always has the post project. The numOutputRows/numOutputVectors metrics are from post project.
|
Run Gluten Clickhouse CI |
|
===== Performance report for TPCH SF2000 with Velox backend, for reference only ====
|
What changes were proposed in this pull request?
This pr adds metrics for sort merge join. Since join always has post project, unfiy the post project metrics for hash join and sort merge join.
How was this patch tested?
Add test and test manually