-
Notifications
You must be signed in to change notification settings - Fork 5.5k
[iceberg] Add metrics reporter for iceberg table scans #24904
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
|
|
@j-sund Can you sign the CLA using the link in the comment above? You can just sign as a personal contributor. Also, before your next push try running |
cfc3890 to
67e5b9b
Compare
ZacBlanco
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just minor things. This looks great!
presto-iceberg/src/main/java/com/facebook/presto/iceberg/RuntimeStatsMetricsReporter.java
Outdated
Show resolved
Hide resolved
presto-iceberg/src/main/java/com/facebook/presto/iceberg/RuntimeStatsMetricsReporter.java
Outdated
Show resolved
Hide resolved
presto-iceberg/src/main/java/com/facebook/presto/iceberg/RuntimeStatsMetricsReporter.java
Outdated
Show resolved
Hide resolved
presto-iceberg/src/main/java/com/facebook/presto/iceberg/RuntimeStatsMetricsReporter.java
Outdated
Show resolved
Hide resolved
presto-iceberg/src/main/java/com/facebook/presto/iceberg/RuntimeStatsMetricsReporter.java
Outdated
Show resolved
Hide resolved
presto-iceberg/src/main/java/com/facebook/presto/iceberg/RuntimeStatsMetricsReporter.java
Outdated
Show resolved
Hide resolved
presto-iceberg/src/main/java/com/facebook/presto/iceberg/RuntimeStatsMetricsReporter.java
Outdated
Show resolved
Hide resolved
presto-iceberg/src/test/java/com/facebook/presto/iceberg/IcebergDistributedSmokeTestBase.java
Outdated
Show resolved
Hide resolved
presto-iceberg/src/test/java/com/facebook/presto/iceberg/IcebergDistributedSmokeTestBase.java
Outdated
Show resolved
Hide resolved
presto-iceberg/src/main/java/com/facebook/presto/iceberg/IcebergSplitManager.java
Show resolved
Hide resolved
72ef7e5 to
6124180
Compare
presto-iceberg/src/main/java/com/facebook/presto/iceberg/IcebergUtil.java
Outdated
Show resolved
Hide resolved
5b96c26 to
8b19e78
Compare
hantangwangd
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This looks like a great feature! Some nits, and a minor problem in the test code.
presto-iceberg/src/main/java/com/facebook/presto/iceberg/RuntimeStatsMetricsReporter.java
Outdated
Show resolved
Hide resolved
presto-iceberg/src/main/java/com/facebook/presto/iceberg/IcebergSplitManager.java
Outdated
Show resolved
Hide resolved
presto-iceberg/src/main/java/com/facebook/presto/iceberg/TableStatisticsMaker.java
Outdated
Show resolved
Hide resolved
presto-iceberg/src/main/java/com/facebook/presto/iceberg/TableStatisticsMaker.java
Outdated
Show resolved
Hide resolved
presto-iceberg/src/test/java/com/facebook/presto/iceberg/IcebergDistributedSmokeTestBase.java
Outdated
Show resolved
Hide resolved
|
I have submit a PR #25140 to make sure that the table name loaded from all catalogs have consistent formatting |
8e4ba6a to
9ef6b65
Compare
ZacBlanco
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just a few style nits. Otherwise LGTM! Can you also squash all of your commits into one?
presto-iceberg/src/main/java/com/facebook/presto/iceberg/IcebergSplitManager.java
Outdated
Show resolved
Hide resolved
presto-iceberg/src/main/java/com/facebook/presto/iceberg/IcebergUtil.java
Outdated
Show resolved
Hide resolved
presto-iceberg/src/main/java/com/facebook/presto/iceberg/IcebergUtil.java
Outdated
Show resolved
Hide resolved
presto-iceberg/src/main/java/com/facebook/presto/iceberg/IcebergUtil.java
Outdated
Show resolved
Hide resolved
ZacBlanco
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM! Can you please squash all of your commits into one? Also, try to write a descriptive commit title+message
bdc7f4b to
ff80c23
Compare
ff80c23 to
55e21d8
Compare
hantangwangd
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the fix, LGTM.
|
@j-sund, there have been some fixes to the CI tests since the last activity on this PR. If you rebase, these pending tests might pass. |
55e21d8 to
f09e109
Compare
|
maven check failure unable to run 'yarn' for presto-ui. not sure if this was just a mishap with the github CI runner? My code shouldn't have affected the presto-ui. Any insight would be appreciated. |
|
Looks like occasional failures, fixed by a rerun. Hi @ZacBlanco, do you have any more comments? If not, I think this PR is ready to be merged. |
A rerun seems to have worked, at this time I'm seeing "All checks have passed / 108 successful checks". |
|
@hantangwangd I wanted to check the status of this long standing pr. Im not sure if Zac is an active iceberg maintainer anymore |
There has been no response for a considerable period of time.
|
@j-sund Sorry for the delay of merging. Following the lazy consensus mentioned by @tdcmeehan here, I dismissed ZacBlanco's review. And will merge the PR tomorrow if there are no further objections. |
Description
Adds a new MetricsReporter implementation to propagate Iceberg table scan metrics into session RuntimeStats
Motivation and Context
Exposing these metrics can improve query observability, which thus makes future improvements to the Iceberg integration a little easier. This also enables better performance monitoring.
Impact
Allows for better visibility into the Iceberg integration.
Test Plan
Ensure the metric is visible and being populated. Basic tests are included in IcebergDistributedSmokeTestBase.java class
Contributor checklist
Release Notes