Skip to content

[feature-wip](parquet-reader) add detail profile for parquet reader#13095

Merged
morningman merged 3 commits intoapache:masterfrom
AshinGau:file-reader
Oct 2, 2022
Merged

[feature-wip](parquet-reader) add detail profile for parquet reader#13095
morningman merged 3 commits intoapache:masterfrom
AshinGau:file-reader

Conversation

@AshinGau
Copy link
Member

@AshinGau AshinGau commented Sep 30, 2022

Proposed changes

Add more detail profile for ParquetReader:
ParquetColumnReadTime: the total time of reading parquet columns
ParquetDecodeDictTime: time to parse dictionary page
ParquetDecodeHeaderTime: time to parse page header
ParquetDecodeLevelTime: time to parse page's definition/repetition level
ParquetDecodeValueTime: time to decode page data into doris column
ParquetDecompressCount: counter of decompressing page data
ParquetDecompressTime: time to decompress page data
ParquetParseMetaTime: time to parse parquet meta data

 ┌──────────────────────────────────────────┐
 │[0: VFILE_SCAN_NODE]                      │
 │(Active: 1s476ms, non-child: 88.00)       │
 │  - Counters:                             │
 │      - FileReadBytes: 73.60 MB           │
 │      - FileReadCalls: 10                 │
 │      - FileReadTime: 866.685ms           │
 │      - MaxScannerThreadNum: 1            │
 │      - NewlyCreateFreeBlocksNum: 93      │
 │      - NumScanners: 1                    │
 │      - ParquetColumnReadTime: 1s503ms    │
 │      - ParquetDecodeDictTime: 0ns        │
 │      - ParquetDecodeHeaderTime: 868.392ms│
 │      - ParquetDecodeLevelTime: 144.826us │
 │      - ParquetDecodeValueTime: 403.950ms │
 │      - ParquetDecompressCount: 305       │
 │      - ParquetDecompressTime: 154.345ms  │
 │      - ParquetFilteredBytes: 0.00        │
 │      - ParquetFilteredGroups: 0          │
 │      - ParquetFilteredRowsByGroup: 0     │
 │      - ParquetFilteredRowsByPage: 0      │
 │      - ParquetParseMetaTime: 119.999ms   │
 │      - ParquetReadBytes: 73.60 MB        │
 │      - ParquetReadGroups: 10             │
 │      - PeakMemoryUsage: 0.00             │
 │      - PreAllocFreeBlocksNum: 17         │
 │      - RowsRead: 6.001215M (6001215)     │
 │      - RowsReturned: 6.001215M (6001215) │
 │      - RowsReturnedRate: 4.064708M /sec  │
 │      - ScannerBatchWaitTime: 1s469ms     │
 │      - ScannerCtxSchedCount: 0           │
 │      - ScannerSchedCount: 0              │
 │      - ScannerWorkerWaitTime: 0ns        │
 │      - TotalReadThroughput: 0            │
 └──────────────────────────────────────────┘

Checklist(Required)

  1. Does it affect the original behavior:
    • Yes
    • No
    • I don't know
  2. Has unit tests been added:
    • Yes
    • No
    • No Need
  3. Has document been added or modified:
    • Yes
    • No
    • No Need
  4. Does it need to update dependencies:
    • Yes
    • No
  5. Are there any changes that cannot be rolled back:
    • Yes (If Yes, please explain WHY)
    • No

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@AshinGau AshinGau force-pushed the file-reader branch 2 times, most recently from 62f552e to 98ab775 Compare October 2, 2022 03:12
Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@morningman morningman merged commit 026ffaf into apache:master Oct 2, 2022
FreeOnePlus pushed a commit to FreeOnePlus/doris that referenced this pull request Oct 8, 2022
…pache#13095)

Add more detail profile for ParquetReader:
ParquetColumnReadTime: the total time of reading parquet columns
ParquetDecodeDictTime: time to parse dictionary page
ParquetDecodeHeaderTime: time to parse page header
ParquetDecodeLevelTime: time to parse page's definition/repetition level
ParquetDecodeValueTime: time to decode page data into doris column
ParquetDecompressCount: counter of decompressing page data
ParquetDecompressTime: time to decompress page data
ParquetParseMetaTime: time to parse parquet meta data
FreeOnePlus pushed a commit to FreeOnePlus/doris that referenced this pull request Oct 8, 2022
…pache#13095)

Add more detail profile for ParquetReader:
ParquetColumnReadTime: the total time of reading parquet columns
ParquetDecodeDictTime: time to parse dictionary page
ParquetDecodeHeaderTime: time to parse page header
ParquetDecodeLevelTime: time to parse page's definition/repetition level
ParquetDecodeValueTime: time to decode page data into doris column
ParquetDecompressCount: counter of decompressing page data
ParquetDecompressTime: time to decompress page data
ParquetParseMetaTime: time to parse parquet meta data
FreeOnePlus pushed a commit to FreeOnePlus/doris that referenced this pull request Oct 8, 2022
…pache#13095)

Add more detail profile for ParquetReader:
ParquetColumnReadTime: the total time of reading parquet columns
ParquetDecodeDictTime: time to parse dictionary page
ParquetDecodeHeaderTime: time to parse page header
ParquetDecodeLevelTime: time to parse page's definition/repetition level
ParquetDecodeValueTime: time to decode page data into doris column
ParquetDecompressCount: counter of decompressing page data
ParquetDecompressTime: time to decompress page data
ParquetParseMetaTime: time to parse parquet meta data
FreeOnePlus pushed a commit to FreeOnePlus/doris that referenced this pull request Oct 8, 2022
…pache#13095)

Add more detail profile for ParquetReader:
ParquetColumnReadTime: the total time of reading parquet columns
ParquetDecodeDictTime: time to parse dictionary page
ParquetDecodeHeaderTime: time to parse page header
ParquetDecodeLevelTime: time to parse page's definition/repetition level
ParquetDecodeValueTime: time to decode page data into doris column
ParquetDecompressCount: counter of decompressing page data
ParquetDecompressTime: time to decompress page data
ParquetParseMetaTime: time to parse parquet meta data
FreeOnePlus pushed a commit to FreeOnePlus/doris that referenced this pull request Oct 8, 2022
…pache#13095)

Add more detail profile for ParquetReader:
ParquetColumnReadTime: the total time of reading parquet columns
ParquetDecodeDictTime: time to parse dictionary page
ParquetDecodeHeaderTime: time to parse page header
ParquetDecodeLevelTime: time to parse page's definition/repetition level
ParquetDecodeValueTime: time to decode page data into doris column
ParquetDecompressCount: counter of decompressing page data
ParquetDecompressTime: time to decompress page data
ParquetParseMetaTime: time to parse parquet meta data
FreeOnePlus pushed a commit to FreeOnePlus/doris that referenced this pull request Oct 8, 2022
…pache#13095)

Add more detail profile for ParquetReader:
ParquetColumnReadTime: the total time of reading parquet columns
ParquetDecodeDictTime: time to parse dictionary page
ParquetDecodeHeaderTime: time to parse page header
ParquetDecodeLevelTime: time to parse page's definition/repetition level
ParquetDecodeValueTime: time to decode page data into doris column
ParquetDecompressCount: counter of decompressing page data
ParquetDecompressTime: time to decompress page data
ParquetParseMetaTime: time to parse parquet meta data
@AshinGau AshinGau deleted the file-reader branch December 20, 2022 07:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants