New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
API: Remove overflow checks in DefaultCounter causing performance issues #8297
API: Remove overflow checks in DefaultCounter causing performance issues #8297
Conversation
@@ -63,7 +63,6 @@ public void increment() { | |||
|
|||
@Override | |||
public void increment(long amount) { | |||
Math.addExact(counter.longValue(), amount); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Calling longValue()
on LongAdd
is expensive as it performs an aggregation over cells. It basically removes all the benefits of LongAdd
that is supposed to scale well in case of contention. We call it every time the counter is being modified. I am also not sure having an overflow (which is unlikely given that we are using longs) should fail queries, for instance. These are metrics and they should add as little overhead as possible.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ctx.scanMetrics().totalFileSizeInBytes().increment(dataFile.fileSizeInBytes()) is the one that had the potential. With long as datatype, it's like 8192 Petabyte data read, so doesn't seem to be an issue as of now :)
Just curious, since it's a public class and if someone is using it, should we mention this in javadoc?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
since it's a public class and if someone is using it, should we mention this in javadoc?
That's a good idea, will do if we decide to proceed with the change.
407c069
to
3beb2a7
Compare
@nastra, could you take a look? |
@@ -54,18 +54,6 @@ public void intCounter() { | |||
Assertions.assertThat(counter.unit()).isEqualTo(MetricsContext.Unit.BYTES); | |||
} | |||
|
|||
@Test |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These counters have been deprecated previously.
@@ -107,7 +106,6 @@ public void increment() { | |||
|
|||
@Override | |||
public void increment(Integer amount) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For IntCounter
, I believe that we can still add a cheap overflow check at the value()
method.
public Integer value() {
long sum = counter.longValue();
if (sum > Integer.MaxValue) {
...
}
return (int) sum
}
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It is the old deprecated counter API that is not used anymore but no harm adding it. Let me do it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for finding this @aokolnychyi. I think we're ok not checking for an overflow. I've checked other frameworks (dropwizard/micrometer) and their counter implementations also don't seem to be doing overflow checks, so this change seems reasonable to me
@@ -29,6 +29,8 @@ public interface Counter { | |||
/** | |||
* Increment the counter by the provided amount. | |||
* | |||
* <p>Implementations may skip the overflow check for better write throughput. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should we add in DefaultCounter that it doesn't check for overflow? Not sure if it's too much.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I see that DefaultCounter
has no other method-level docs so I'd probably just stick to a note in the public interface for now.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1 LGTM
Thanks @aokolnychyi
b6d9c20
to
0be51ce
Compare
Thank you, @mystic-lama @advancedxy @nastra! |
This PR removes the overflow check in
DefaultCounter
, which causes performance issues when the counter is being accessed from multiple threads.Prior to this PR:
After this PR: