Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FEATURE][SPARK] Collect and summarize the executorRunTime and executorCpuTime of the statement #6107

Closed
3 of 4 tasks
cxzl25 opened this issue Feb 28, 2024 · 1 comment
Closed
3 of 4 tasks

[FEATURE][SPARK] Collect and summarize the executorRunTime and executorCpuTime of the statement #6107

cxzl25 opened this issue Feb 28, 2024 · 1 comment
Assignees
@XorSum
Labels
kind:feature Feature request priority:major

Comments

@cxzl25
Copy link
Contributor

cxzl25 commented Feb 28, 2024 •
edited

Code of Conduct

Search before asking

  • I have searched in the issues and found no similar issues.

Describe the feature

By aggregating the statement's taskMetrics.executorRunTime and taskMetrics.executorCpuTime to collect the CPU time consumed by the statement, a rough SQL-level billing can be achieved.

Motivation

No response

Describe the solution

No response

Additional context

No response

Are you willing to submit PR?

  • Yes. I would be willing to submit a PR with guidance from the Kyuubi community to improve.
  • No. I cannot submit a PR at this time.
@cxzl25 cxzl25 mentioned this issue Feb 28, 2024
Closed
4 tasks
@cxzl25 cxzl25 changed the title [FEATURE][SPARK] Collect and summarize the executorRunTime and executorCpuTime of the statement [FEATURE][SPARK] Collect and summarize the executorRunTime and executorCpuTime of the statement Feb 28, 2024
@XorSum
Copy link
Contributor

XorSum commented Feb 28, 2024

please assign to me

@XorSum XorSum mentioned this issue Feb 28, 2024
Closed
4 tasks
@cxzl25 cxzl25 closed this as completed in 3bc28fd Mar 4, 2024
cxzl25 pushed a commit that referenced this issue Mar 4, 2024
@XorSum @cxzl25
[KYUUBI #6107][FOLLOWUP] Add comments for the newly added parameters …
a9b90c7 
…of the Spark event

# 🔍 Description
## Issue References 🔗

This pull request fixes #6112 (comment)

## Describe Your Solution 🔧

add comments for the newly added parameters

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

#### Related Unit Tests

---

# Checklist 📝

- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #6124 from XorSum/features/features/spark-engine-cpu-time-collect-comments.

Closes #6107

d93028b [bkhan] add comments

Authored-by: bkhan <bkhan@trip.com>
Signed-off-by: Shaoyun Chen <csy@apache.org>
zhaohehuhu pushed a commit to zhaohehuhu/incubator-kyuubi that referenced this issue Mar 21, 2024
@XorSum @zhaohehuhu
[KYUUBI apache#6107] [Spark] Collect and summarize the `executorRunTi…
017c4ba 
…me` and `executorCpuTime` of the statement

# 🔍 Description
## Issue References 🔗

This pull request fixes apache#6107

## Describe Your Solution 🔧

The total execution time of a statement (or a session) is the summary of the execution time of the stages belonging to the statement (or session).
The total execution time of a stage is collected from `SQLOperationListener#onStageCompleted`.
The total execution times of the statement or a session are stored in the engine events or output to the log.

<img width="962" alt="截屏2024-02-29 14 47 50" src="https://github.com/apache/kyuubi/assets/23011702/176df1db-bb20-428b-94b8-fa02c946fde2">
<img width="1143" alt="截屏2024-02-29 14 47 21" src="https://github.com/apache/kyuubi/assets/23011702/8cfc6a72-f6e8-45b6-bdda-30296c94c893">

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

#### Related Unit Tests

---

# Checklist 📝

- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes apache#6112 from XorSum/features/spark-engine-cpu-time-collect.

Closes apache#6107

8028005 [bkhan] check same group
d9efa2d [bkhan] formatDuration
a8841cd [bkhan] update
2507159 [bkhan] Apply suggestions from code review
cfed2b9 [bkhan] use formatDurationVerbose
444d4aa [bkhan] Collect and summarize the executorRunTime and executorCpuTime of the statement

Authored-by: bkhan <bkhan@trip.com>
Signed-off-by: Shaoyun Chen <csy@apache.org>
zhaohehuhu pushed a commit to zhaohehuhu/incubator-kyuubi that referenced this issue Mar 21, 2024
@XorSum @zhaohehuhu
[KYUUBI apache#6107][FOLLOWUP] Add comments for the newly added param…
69dc4b6 
…eters of the Spark event

# 🔍 Description
## Issue References 🔗

This pull request fixes apache#6112 (comment)

## Describe Your Solution 🔧

add comments for the newly added parameters

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

#### Related Unit Tests

---

# Checklist 📝

- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes apache#6124 from XorSum/features/features/spark-engine-cpu-time-collect-comments.

Closes apache#6107

d93028b [bkhan] add comments

Authored-by: bkhan <bkhan@trip.com>
Signed-off-by: Shaoyun Chen <csy@apache.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@XorSum XorSum

Labels
kind:feature Feature request priority:major
Projects
None yet
Milestone
No milestone
2 participants
@cxzl25 @XorSum