-
Notifications
You must be signed in to change notification settings - Fork 1.8k
Add opening, scanning, processing metrics in file stream #3070
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
| partition_values, | ||
| } => match ready!(reader.poll_next_unpin(cx)) { | ||
| Some(result) => { | ||
| if let Some(instant) = |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Here will add first one batch processing time in total scanning time, I think it's acceptable🤔
Codecov Report
@@ Coverage Diff @@
## master #3070 +/- ##
==========================================
+ Coverage 85.85% 85.93% +0.08%
==========================================
Files 289 289
Lines 51890 52118 +228
==========================================
+ Hits 44548 44789 +241
+ Misses 7342 7329 -13
📣 Codecov can now indicate which changes are the most critical in Pull Requests. Learn more |
|
@alamb PTAL😊 |
alamb
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice work @Ted-Jiang -- I think this could go in as is.
One thought I had was that you might consider wrapping the logic for Time / Instant handling into its own struct
So instead of
pub time_opening: (metrics::Time, Option<Instant>),You could have something like
pub time_opening: StartableTimerwhere
struct StartableTime {
metrics::Time,
start: Option<Instant>
}
impl StartableTime {
fn start(&mut self) {
assert!(self.start.is_none());
self.start = Some(Instant::now());
}
fn stop(&mut self) {
if let Some(start) = self.start.take() {
self.timer.add_elapsed(start);
}
}| .time_scanning | ||
| .0 | ||
| .add_elapsed(instant); | ||
| } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I wonder does it matter if we update these metrics on plan failure? As in if the stream is never ready the 'file opening time' will never get updated. Maybe that is ok 🤔
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes, if the stream is never ready, it just not the metric issue😂
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
|
@alamb Thanks for your kindly advice! 👍 |
|
Thanks @Ted-Jiang -- great to hear -- looks like the new assertions have been triggered in some CI tests. Testing for the win! |
| Limit, | ||
| } | ||
|
|
||
| struct StartableTime { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍
| None => return Poll::Ready(None), | ||
| }; | ||
|
|
||
| self.file_stream_metrics.time_opening.start(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👌 looking very nice
| } | ||
| None => self.state = FileStreamState::Idle, | ||
| None => { | ||
| self.file_stream_metrics.time_scanning.stop(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Forgot stop timer here 😂
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Testing for the Win!
|
Thanks again @Ted-Jiang |
|
Benchmark runs are scheduled for baseline = 31381bf and contender = 098f0b0. 098f0b0 is a master commit associated with this PR. Results will be available as each benchmark for each run completes. |
Which issue does this PR close?
Closes #3024 .
Rationale for this change
What changes are included in this PR?
Are there any user-facing changes?