-
Notifications
You must be signed in to change notification settings - Fork 2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Core: Introduce CompositeMetricsReporter #7337
Core: Introduce CompositeMetricsReporter #7337
Conversation
e08d9b6
to
dbf7181
Compare
@@ -448,4 +475,46 @@ public static MetricsReporter loadMetricsReporter(Map<String, String> properties | |||
|
|||
return reporter; | |||
} | |||
|
|||
static class CompositeMetricsReporter implements MetricsReporter { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@rdblue as we discussed I've made this an inner class. However, we could consider moving the class out of CatalogUtil
and replace https://github.com/apache/iceberg/blob/master/core/src/main/java/org/apache/iceberg/TableScanContext.java#L359-L366 with a CompositeMetricsReporter
.
For cases where we'd like to use multiple reporters during scans
TableScan tableScan = table
.newScan()
.metricsReporter(...)
.metricsReporter(...)
.metricsReporter(...);
we would internally instantiate a CompositeMetricsReporter
in TableScanContext#reportWith(..)
and then have TableScanContext#metricsReporter()
return MetricsReporter
rather than Collection<MetricsReporter>
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
dbf7181
to
7d276c5
Compare
Why do we need to support multiple reporters? I don't think we need to change the behavior. |
7d276c5
to
af1440f
Compare
13d7859
to
74c3615
Compare
core/src/main/java/org/apache/iceberg/metrics/CompositeMetricsReporter.java
Outdated
Show resolved
Hide resolved
core/src/main/java/org/apache/iceberg/metrics/CompositeMetricsReporter.java
Outdated
Show resolved
Hide resolved
core/src/main/java/org/apache/iceberg/metrics/CompositeMetricsReporter.java
Outdated
Show resolved
Hide resolved
core/src/main/java/org/apache/iceberg/metrics/CompositeMetricsReporter.java
Outdated
Show resolved
Hide resolved
reporter -> { | ||
try { | ||
reporter.report(report); | ||
} catch (Exception e) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this should be RuntimeException
since report
doesn't throw a checked exception.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
the reason I'm catching an Exception
here is that I think we generally don't want to interfere with actual scans or commits when anything on the reporting side fails. Generally speaking, every reporter should make sure that it doesn't interefere with the scan/commit path when reporting and something fails but I'm not sure we can enforce that.
I'm a bit worried that if we only catch RuntimeException
and some reporter misbehaves in an unexpected way, we'd have to go back and fix either the misbehaving reporter or this catch clause.
thoughts?
74c3615
to
8bb4721
Compare
core/src/main/java/org/apache/iceberg/metrics/MetricsReporters.java
Outdated
Show resolved
Hide resolved
core/src/main/java/org/apache/iceberg/metrics/MetricsReporters.java
Outdated
Show resolved
Hide resolved
core/src/main/java/org/apache/iceberg/metrics/MetricsReporters.java
Outdated
Show resolved
Hide resolved
core/src/main/java/org/apache/iceberg/metrics/MetricsReporters.java
Outdated
Show resolved
Hide resolved
db694d4
to
abd5453
Compare
core/src/main/java/org/apache/iceberg/metrics/MetricsReporters.java
Outdated
Show resolved
Hide resolved
core/src/test/java/org/apache/iceberg/metrics/TestMetricsReporters.java
Outdated
Show resolved
Hide resolved
Looks good. Just one minor thing to fix with the sets. |
This also switches from a collection of reporters to the `CompositeMetricsReporter` in `TableScanContext` to fix an issue where duplicate reporters were not handled.
abd5453
to
1d49a8d
Compare
This also switches from a collection of reporters to the `CompositeMetricsReporter` in `TableScanContext` to fix an issue where duplicate reporters were not handled.
This introduces a
CompositeMetricsReporter
that allows registering multiple (simple)MetricsReporter
instances that all get notified whenever a newMetricsReport
is available.This also switches from a collection of reporters to the
CompositeMetricsReporter
inTableScanContext
to fix an issue where duplicate reporters were not handled (see #7337 (comment) for details).