Skip to content

[improvement](profile) add detail memory counter for exec nodes#14806

Merged
yiguolei merged 2 commits intoapache:masterfrom
jacktengg:improve-memory-statistics
Dec 5, 2022
Merged

[improvement](profile) add detail memory counter for exec nodes#14806
yiguolei merged 2 commits intoapache:masterfrom
jacktengg:improve-memory-statistics

Conversation

@jacktengg
Copy link
Contributor

Proposed changes

Issue Number: close #xxx

Problem summary

Add memory usage detail counters, e.g.:
scan node:

VNewOlapScanNode(ods_e3_order_goods_di) (id=0):(Active: 710.812ms, non-child: 4.08%)
              MemoryUsage:
             - FreeBlocks: 814.77 MB
             - QueuedBlocks: 813.66 MB

hash join node:

      VHASH_JOIN_NODE (id=2):(Active: 6s450ms, non-child: 32.27%)
        MemoryUsage:                                                                                                           
           - BuildBlocks: 433.54 MB
           - BuildKeyArena: 0
           - HashTable: 8.00 MB
           - ProbeKeyArena: 0

agg node:

    VAGGREGATION_NODE (id=1):(Active: 917.409ms, non-child: 97.79%)  
      MemoryUsage:
         - HashTable: 168.00 B
         - SerializeKeyArena: 1.55 MB          

sort node:

    VSORT_NODE (id=7):(Active: 30s585ms, non-child: 44.83%) 
       - TOP-N: false
       - MemoryOnlyTrackAllocNoConsiderFree: 22.56 GB
       ... ...
      MemoryUsage:
         - SortBlocks: 10.36 GB

analytic node:

  VANALYTIC_EVAL_NODE (id=8):(Active: 13s362ms, non-child: 0.00%)
     - EvaluationTime: 13.954ms
     - MemoryOnlyTrackAllocNoConsiderFree: 84.94 GB
    ... ...
    MemoryUsage:                                                                 
       - Blocks: 11.75 MB

Checklist(Required)

  1. Does it affect the original behavior:
    • Yes
    • No
    • I don't know
  2. Has unit tests been added:
    • Yes
    • No
    • No Need
  3. Has document been added or modified:
    • Yes
    • No
    • No Need
  4. Does it need to update dependencies:
    • Yes
    • No
  5. Are there any changes that cannot be rolled back:
    • Yes (If Yes, please explain WHY)
    • No

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@github-actions
Copy link
Contributor

github-actions bot commented Dec 4, 2022

clang-tidy review says "All clean, LGTM! 👍"

@hello-stephen
Copy link
Contributor

hello-stephen commented Dec 4, 2022

TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 34.44 seconds
load time: 431 seconds
storage size: 17123356296 Bytes
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221205021624_clickbench_pr_57588.html

@jacktengg jacktengg force-pushed the improve-memory-statistics branch from 25a1db2 to f30fdb8 Compare December 5, 2022 01:55
@github-actions
Copy link
Contributor

github-actions bot commented Dec 5, 2022

clang-tidy review says "All clean, LGTM! 👍"

Copy link
Contributor

@yiguolei yiguolei left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@yiguolei
Copy link
Contributor

yiguolei commented Dec 5, 2022

Just merge it and then we could improve it step by step.

@yiguolei yiguolei merged commit 8c0e13a into apache:master Dec 5, 2022
@github-actions
Copy link
Contributor

github-actions bot commented Dec 5, 2022

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 5, 2022
@github-actions
Copy link
Contributor

github-actions bot commented Dec 5, 2022

PR approved by anyone and no changes requested.

@jacktengg jacktengg deleted the improve-memory-statistics branch December 7, 2022 03:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. area/vectorization reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants