Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add processor level profiling #4238

Closed
BohuTANG opened this issue Feb 24, 2022 · 4 comments
Closed

Add processor level profiling #4238

BohuTANG opened this issue Feb 24, 2022 · 4 comments
Assignees
Labels
A-query Area: databend query

Comments

@BohuTANG
Copy link
Member

BohuTANG commented Feb 24, 2022

Summary

To measure the performance of every processor, the profiles including:

  • How much time each processor cost
  • The total rows before processing
  • The total rows after processing
  • The total bytes before processing
  • The total bytes after processing

Note:
We have did some work related to the processor profiling in the context query level, it's not an elegant solution.
E.g. get DALMetrics from query context, this should get from the Source Processor.

Example

select count(YEAR), avg(ARRTIME) from ontime group by year;

Source Processor ----> Aggregate Processor --> Sink Processor

Source Processor ⇢

  • Attributes: [Full table name, Columns (2 / 109)]
  • Statistics: [Scan progress, Bytes scanned, Percentage scanned from cache, Partitions scanned, Partitions total]
  • Profile Overview: [Processing, Local Disk I/O, Remote Disk I/O]

⇢ Aggregate Processor ⇢

  • Attributes: [Aggregate Functions, Grouping Keys]
  • Profile Overview: [Processing]

⇢ Sink Processor

  • Attributes: [List of Expressions]

TODO

  1. What's the format of all processor profile data (including DAG)?
  2. How to get all the processor profile datas from the query context?
@BohuTANG
Copy link
Member Author

BohuTANG commented Feb 24, 2022

Feel free to comment if have any thoughts @zhang2014

@BohuTANG BohuTANG changed the title Add a global perf counter for query context Add processor level profiling Feb 24, 2022
@BohuTANG BohuTANG added the A-query Area: databend query label Feb 24, 2022
@zhang2014 zhang2014 self-assigned this Feb 24, 2022
@Xuanwo Xuanwo added this to the v0.8 milestone May 20, 2022
@Xuanwo Xuanwo removed this from the v0.8 milestone Aug 15, 2022
@FANNG1
Copy link
Collaborator

FANNG1 commented Sep 14, 2022

is there any progress?

@zhang2014
Copy link
Member

is there any progress?

In progress. It's depend some refactor(we need to hold pipeline executor in query ctx).

@BohuTANG
Copy link
Member Author

#13637

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-query Area: databend query
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants