HBASE-26809: Report client backoff time for server overloaded #4786
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
Looking into the issues above. May be due to my branch being a little behind the branch-2 latest changes. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
briaugenreich
force-pushed
the
HBASE-26809_branch-2
branch
from
ac21df9
to
13728dc
Compare
briaugenreich
force-pushed
the
HBASE-26809_branch-2
branch
from
13728dc
to
140878e
Compare
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
I'm trying to reproduce/troubleshoot the issues above locally. I am able to successfully build with a subset of tests with the following command: When I try to run a spotbugs check locally I get the following error Any tips on troubleshooting my build issues are much appreciated. cc: @bbeaudreault |
|
Actually I think I just found the links to the results on the ci server which is helpful. Though still curious if I can run something locally to test beforehand. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
Everything looks good here, but not sure what to do for existing spotbugs failures. I don't think I introduced the errors called out in the report. cc: @bbeaudreault |
|
Yea, the spotbugs issues look related to the recent spotbugs version upgrade. I think they will be tackled in https://issues.apache.org/jira/browse/HBASE-27373. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
@bbeaudreault Should I rerun this build? It looks like there were some changes merged to fix the spotbugs warnings (https://issues.apache.org/jira/browse/HBASE-27373) |
|
🎊 +1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
May retrigger this latest build... ran the failing test locally and it seemed happy. |
|
Actually, doesn't look like I have permission to do this. @bbeaudreault Should/can I request access to the build server? |
|
Unfortunately only committers have access to the build server. I kicked off a rebuild last night. The easiest way for you to self-service this is to push a commit. Sometimes what I used to do is |
briaugenreich
force-pushed
the
HBASE-26809_branch-2
branch
from
4479504
to
45ad745
Compare
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
The build looks happy. Ready for review. |
hbase-client/src/main/java/org/apache/hadoop/hbase/client/RpcRetryingCallerImpl.java
Show resolved
Hide resolved
hbase-client/src/main/java/org/apache/hadoop/hbase/client/RpcRetryingCallerFactory.java
Show resolved
Hide resolved
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
hmm not sure why the build was aborted, but I can kick off another. |
bbeaudreault
reviewed
Oct 3, 2022
| @@ -65,26 +68,29 @@ public <T> RpcRetryingCaller<T> newCaller() { | |||
| // is cheap as it does not require parsing a complex structure. | |||
| return new RpcRetryingCallerImpl<>(connectionConf.getPauseMillis(), | |||
| connectionConf.getPauseMillisForServerOverloaded(), connectionConf.getRetriesNumber(), | |||
| interceptor, startLogErrorsCnt, connectionConf.getRpcTimeout()); | |||
| interceptor, startLogErrorsCnt, connectionConf.getRpcTimeout(), metrics); | |||
| } | |||
|
|
|||
| public static RpcRetryingCallerFactory instantiate(Configuration configuration) { | |||
There was a problem hiding this comment.
Ok, sorry for the delay here but I've been meaning to look into the usages of these methods. We basically want to ensure that our metrics object is getting in there as often as possible.
So I opened up branch-2 in intellij and looked at the usages of these instantiate overloads:
instantiate(Configuration)-- all of the callers to this method have a ClusterConnection in the call context, which we can pull our metrics from (connection.getConnectionMetrics())- `instantiate(Configuration, ServerStatisticTracker) -- all but the SecureBulkLoadClient and LoadIncrementalHFiles have similar.
So I think we should try to add ConnectionMetrics to all of the methods here, and we can pass null in for the 2 cases above.
|
💔 -1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
bbeaudreault
pushed a commit
that referenced
this pull request
Nov 15, 2022
Co-authored-by: Briana Augenreich <baugenreich@hubspot.com> Signed-off-by: Bryan Beaudreault <bbeaudreault@apache.org>
bbeaudreault
pushed a commit
to HubSpot/hbase
that referenced
this pull request
Nov 18, 2022
…overloaded (apache#4786) Co-authored-by: Briana Augenreich <baugenreich@hubspot.com> Signed-off-by: Bryan Beaudreault <bbeaudreault@apache.org>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Branch 2 changes for #4729
Enable metric reporting on backoff time when a server is overloaded.