Skip to content

Comments

[Feature](profile) support SHOW EXPORT PROFILE stmt.#15957

Closed
dutyu wants to merge 5 commits intoapache:masterfrom
dutyu:support-show-export-profile
Closed

[Feature](profile) support SHOW EXPORT PROFILE stmt.#15957
dutyu wants to merge 5 commits intoapache:masterfrom
dutyu:support-show-export-profile

Conversation

@dutyu
Copy link
Contributor

@dutyu dutyu commented Jan 16, 2023

Proposed changes

Add SHOW EXPORT PROFILE statement.

mysql> show export profile '/';
+-------+-----------------------------------+------+-----------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------+---------------------+---------------------+-----------+------------+
| JobId | QueryId                           | User | DefaultDb | SQL                                                                                                                                                                                                                                     | QueryType | StartTime           | EndTime             | TotalTime | QueryState |
+-------+-----------------------------------+------+-----------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------+---------------------+---------------------+-----------+------------+
| 17007 | 4e174a17ce7d4973-8d5cc25dfc786325 | xxx  | 10341     | EXPORT TABLE `default_cluster:test_d`.`t1`
 TO '/tmp/t1'
PROPERTIES ("exec_mem_limit" = "2147483648", "tablet_num_per_task" = "5", "label" = "export_de2b89b3-11ab-4175-be4e-e728e717486f", "timeout" = "7200")
 WITH BROKER 'local' () | Export    | 2023-01-16 13:30:00 | 2023-01-16 13:30:08 | 7s555ms   | FINISHED   |
+-------+-----------------------------------+------+-----------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------+---------------------+---------------------+-----------+------------+
1 row in set (0.04 sec)
mysql> show export profile '/17007';
+-----------------------------------+------------+
| TaskId                            | ActiveTime |
+-----------------------------------+------------+
| 51d5047016174b3a-9fdd2beee7d9e386 | 7s504ms    |
+-----------------------------------+------------+
1 row in set (0.01 sec)
mysql> show export profile '/17007/51d5047016174b3a-9fdd2beee7d9e386';
+-----------------------------------+------------------+------------+
| Instances                         | Host             | ActiveTime |
+-----------------------------------+------------------+------------+
| 51d5047016174b3a-9fdd2beee7d9e387 | 10.94.22.11:9060 | 1s566ms    |
+-----------------------------------+------------------+------------+
1 row in set (0.01 sec)
mysql> show export profile '/17007/51d5047016174b3a-9fdd2beee7d9e386/51d5047016174b3a-9fdd2beee7d9e387'\G
*************************** 1. row ***************************
Instance:
   ┌───────────────────────────────────────────┐
   │[-1: ExportSink]                           │
   │(Active: 977.874ms, non-child: 0.00)       │
   │  - Counters:                              │
   │      - BytesExported: 32.19 MB            │
   │      - RowsExported: 16.877216M (16877216)│
   │      - WriteTime: 126.116ms               │
   └───────────────────────────────────────────┘
                         │
                         │
┌─────────────────────────────────────────────────┐
│[0: OLAP_SCAN_NODE]                              │
│(Active: 573.945ms, non-child: 0.00)             │
│  - Info:                                        │
│      - Table: t1                                │
│      - KeyRanges: ScanKeys:ScanKey=[null() : �] │
│      - PushdownPredicate: []                    │
│  - Counters:                                    │
│      - BatchQueueWaitTime: 562.699ms            │
│      - BytesRead: 259.54 MB                     │
│      - MemoryOnlyTrackAllocNoConsiderFree: 0.00 │
│      - NumDiskAccess: 1                         │
│      - NumScanners: 1                           │
│      - PeakMemoryUsage: 0.00                    │
│      - ProjectionTime: 0ns                      │
│      - RowsRead: 16.877216M (16877216)          │
│      - RowsReturned: 16.877216M (16877216)      │
│      - RowsReturnedRate: 29.405632M /sec        │
│      - ScannerBatchWaitTime: 1s586ms            │
│      - ScannerSchedCount : 1.031K (1031)        │
│      - ScannerWorkerWaitTime: 34.105ms          │
│      - TabletCount : 1                          │
│      - TotalReadThroughput: 0.0 /sec            │
└─────────────────────────────────────────────────┘
                        ┌┘
                        │
      ┌───────────────────────────────────┐
      │[OlapScanner(t1)]                  │
      │(Active: 0ns, non-child: 0.00)     │
      │  - Counters:                      │
      │      - BlockConvertTime: 154.720ms│
      │      - BlockFetchTime: 427.532ms  │
      │      - ReaderInitTime: 7.600ms    │
      │      - RowsDelFiltered: 0         │
      │      - RowsPushedCondFiltered: 0  │
      │      - ScanCpuTime: 1s479ms       │
      │      - ScanTime: 1s468ms          │
      │      - ShowHintsTime_V1: 0ns      │
      └───────────────────────────────────┘
                        └┐
                         │
   ┌──────────────────────────────────────────┐
   │[SegmentIterator]                         │
   │(Active: 0ns, non-child: 0.00)            │
   │  - Counters:                             │
   │      - BitmapIndexFilterTimer: 2.0us     │
   │      - BlockLoadTime: 260.573ms          │
   │      - BlockSeekCount: 0                 │
   │      - BlockSeekTime: 0ns                │
   │      - BlocksLoad: 16.482K (16482)       │
   │      - CachedPagesNum: 1.035K (1035)     │
   │      - CompressedBytesRead: 0.00         │
   │      - DecompressorTimer: 0ns            │
   │      - FirstReadTime: 251.65ms           │
   │      - IOTimer: 0ns                      │
   │      - IndexLoadTime_V1: 0ns             │
   │      - LazyReadTime: 0ns                 │
   │      - NumSegmentFiltered: 0             │
   │      - NumSegmentTotal: 2                │
   │      - OutputColumnTime: 0ns             │
   │      - RawRowsRead: 16.877216M (16877216)│
   │      - RowsBitmapIndexFiltered: 0        │
   │      - RowsBloomFilterFiltered: 0        │
   │      - RowsConditionsFiltered: 0         │
   │      - RowsKeyRangeFiltered: 0           │
   │      - RowsStatsFiltered: 0              │
   │      - RowsVectorPredFiltered: 0         │
   │      - ShortPredEvalTime: 0ns            │
   │      - TotalPagesNum: 1.035K (1035)      │
   │      - UncompressedBytesRead: 0.00       │
   │      - VectorPredEvalTime: 1.811ms       │
   └──────────────────────────────────────────┘

1 row in set (0.00 sec)

Problem summary

Describe your changes.

Checklist(Required)

  1. Does it affect the original behavior:
    • Yes
    • No
    • I don't know
  2. Has unit tests been added:
    • Yes
    • No
    • No Need
  3. Has document been added or modified:
    • Yes
    • No
    • No Need
  4. Does it need to update dependencies:
    • Yes
    • No
  5. Are there any changes that cannot be rolled back:
    • Yes (If Yes, please explain WHY)
    • No

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@github-actions github-actions bot added area/planner Issues or PRs related to the query planner area/load Issues or PRs related to all kinds of load labels Jan 16, 2023
@hello-stephen
Copy link
Contributor

hello-stephen commented Jan 16, 2023

TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 33.83 seconds
load time: 467 seconds
storage size: 17170807246 Bytes
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230206042338_clickbench_pr_90619.html

dutyu pushed a commit to dutyu/doris that referenced this pull request Jan 17, 2023
@dutyu dutyu force-pushed the support-show-export-profile branch from 86cf512 to cb153c2 Compare February 2, 2023 06:03
dutyu pushed a commit to dutyu/doris that referenced this pull request Feb 2, 2023
@dutyu dutyu force-pushed the support-show-export-profile branch from cb153c2 to 1e6ce12 Compare February 6, 2023 03:26
@dutyu dutyu closed this May 8, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/load Issues or PRs related to all kinds of load area/planner Issues or PRs related to the query planner

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants