Skip to content

(enhancement)(cache) Add remote segment cache statistics to profile#12597

Closed
zxealous wants to merge 2 commits intoapache:masterfrom
zxealous:add-cache-statistics
Closed

(enhancement)(cache) Add remote segment cache statistics to profile#12597
zxealous wants to merge 2 commits intoapache:masterfrom
zxealous:add-cache-statistics

Conversation

@zxealous
Copy link
Contributor

@zxealous zxealous commented Sep 14, 2022

Proposed changes

Issue Number: close #12598

Problem summary

Add some segment cache statistics, for example:

Query:
    Summary:
          -  Start  Time:  2022-09-14  15:18:27
          -  End  Time:  2022-09-14  15:18:28
          -  Total:  1s339ms
          -  Query  State:  EOF
          -  Query  ID:  3042c4b738594677-b8e267fb3d97ea93
          -  Query  Type:  Query
          -  Doris  Version:  trunk
          -  User:  root
          -  Default  Db:  default_cluster:test
          -  Sql  Statement:  select  *  from  table_s3
          -  Is  Cached:  No
          -  Trace  ID:  
        Execution  Summary:
              -  Analysis  Time:  17.768ms
              -  Plan  Time:  75.774ms
              -  Schedule  Time:  845.839ms
              -  Wait  and  Fetch  Result  Time:  379.989ms
        Segment  Cache  Statistics:
              -  ReadSegmentCacheFilesNum:  11					// New Add
              -  DownloadSegmentCacheFilesNum:  3				// New Add
              -  HitSegmentCacheFilesNum:  8					// New Add
              -  HitSegmentCacheFilesPercent:  72.73%				// New Add
    Execution  Profile  3042c4b738594677-b8e267fb3d97ea93:(Active:  1s340ms,  %  non-child:  100.00%)
        Fragment  0:
            Instance  3042c4b738594677-b8e267fb3d97ea95  (host=TNetworkAddress(hostname:)):(Active:  377.112ms,  %  non-child:  0.07%)
                  -  FragmentCpuTime:  1.196ms
                  -  FragmentDownloadSegmentCacheFileNum:  0			// New Add
                  -  FragmentHitSegmentCacheFileNum:  0				// New Add
                  -  FragmentReadSegmentCacheFileNum:  0			// New Add
                  -  MemoryLimit:  2.00  GB
                  -  PeakMemoryUsage:  0.00  
                  -  RowsProduced:  11
                BlockMgr:
                      -  ...
                VDataBufferSender  (dst_fragment_instance_id=3042c4b738594677--471d9804c268156b):
                      -  ...
                VEXCHANGE_NODE  (id=1):(Active:  376.140ms,  %  non-child:  28.06%)
                      -  ...
        Fragment  1:
            Instance  3042c4b738594677-b8e267fb3d97ea94  (host=TNetworkAddress(hostname:)):(Active:  377.266ms,  %  non-child:  0.08%)
                  -  FragmentCpuTime:  1.150ms
                  -  FragmentDownloadSegmentCacheFileNum:  3			// New Add
                  -  FragmentHitSegmentCacheFileNum:  8				// New Add
                  -  FragmentReadSegmentCacheFileNum:  11			// New Add
                  -  MemoryLimit:  2.00  GB
                  -  PeakMemoryUsage:  6.81  MB
                  -  RowsProduced:  11
                BlockMgr:
                      -  ...
                VDataStreamSender  (dst_id=1,  dst_fragments=[{"3042c4b738594677-b8e267fb3d97ea95"}]):(Active:  400.700us,  %  non-child:  0.03%)
                      -  ... 
                VOLAP_SCAN_NODE  (id=0):(Active:  375.735ms,  %  non-child:  28.03%)
                      -  ...
                      -  DownloadSegmentsNum:  3				// New Add
                      -  HitCacheSegmentsNum:  8				// New Add
                      -  ... 
                      -  ReadSegmentsNum:  11					// New Add
                      -  ...
                    VOlapScanner(table_s3):
                          -  ...
                        SegmentIterator:
                              -  ...

Checklist(Required)

  1. Does it affect the original behavior:
    • Yes
    • No
    • I don't know
  2. Has unit tests been added:
    • Yes
    • No
    • No Need
  3. Has document been added or modified:
    • Yes
    • No
    • No Need
  4. Does it need to update dependencies:
    • Yes
    • No
  5. Are there any changes that cannot be rolled back:
    • Yes (If Yes, please explain WHY)
    • No

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@zxealous zxealous force-pushed the add-cache-statistics branch from 496b68a to 1000bfb Compare September 14, 2022 07:46
cache_path, config::file_cache_alive_time_sec, file_reader,
config::file_cache_type);
segment->_file_reader = cache_reader;
cache_reader->init_cache_stats(is_query, stats);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Segment is reused, stats need to be reset to 0 for each query.

@zxealous zxealous closed this Sep 26, 2022
@zxealous zxealous deleted the add-cache-statistics branch February 22, 2023 11:08
@zxealous zxealous restored the add-cache-statistics branch February 22, 2023 11:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Enhancement](cache) Add remote segment cache statistics to profile

2 participants