Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Avoid logging excessive exceptions in ProxiedResource #6229

Merged
merged 1 commit into from Aug 2, 2019

Conversation

@bernd
Copy link
Member

commented Jul 31, 2019

Connection errors in ProxiedResource can happen if other nodes in the cluster are restarting. Only log the stack trace if DEBUG is enabled for the logger. Otherwise log a one-line message.

Example stack trace
2019-07-31 23:37:03,022 WARN : org.graylog2.shared.rest.resources.ProxiedResource - Unable to call http://127.0.0.1:9010/api/system/metrics/multiple on node <38fe132d-0a9a-4194-ae18-73d9fd1e134c>
java.net.ConnectException: Failed to connect to /127.0.0.1:9010
	at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.java:265) ~[okhttp-3.14.2.jar:?]
	at okhttp3.internal.connection.RealConnection.connect(RealConnection.java:183) ~[okhttp-3.14.2.jar:?]
	at okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.java:224) ~[okhttp-3.14.2.jar:?]
	at okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.java:108) ~[okhttp-3.14.2.jar:?]
	at okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.java:88) ~[okhttp-3.14.2.jar:?]
	at okhttp3.internal.connection.Transmitter.newExchange(Transmitter.java:169) ~[okhttp-3.14.2.jar:?]
	at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:41) ~[okhttp-3.14.2.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[okhttp-3.14.2.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[okhttp-3.14.2.jar:?]
	at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:94) ~[okhttp-3.14.2.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[okhttp-3.14.2.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[okhttp-3.14.2.jar:?]
	at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) ~[okhttp-3.14.2.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[okhttp-3.14.2.jar:?]
	at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:88) ~[okhttp-3.14.2.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[okhttp-3.14.2.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[okhttp-3.14.2.jar:?]
	at org.graylog2.rest.RemoteInterfaceProvider.lambda$get$0(RemoteInterfaceProvider.java:61) ~[classes/:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[okhttp-3.14.2.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[okhttp-3.14.2.jar:?]
	at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:221) ~[okhttp-3.14.2.jar:?]
	at okhttp3.RealCall.execute(RealCall.java:81) ~[okhttp-3.14.2.jar:?]
	at retrofit2.OkHttpCall.execute(OkHttpCall.java:188) ~[retrofit-2.6.0.jar:?]
	at org.graylog2.shared.rest.resources.ProxiedResource.lambda$null$0(ProxiedResource.java:78) ~[classes/:?]
	at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266) [?:1.8.0_222]
	at java.util.concurrent.FutureTask.run(FutureTask.java) [?:1.8.0_222]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_222]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_222]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_222]
Caused by: java.net.ConnectException: Connection refused (Connection refused)
	at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_222]
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_222]
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_222]
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_222]
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_222]
	at java.net.Socket.connect(Socket.java:589) ~[?:1.8.0_222]
	at okhttp3.internal.platform.Platform.connectSocket(Platform.java:130) ~[okhttp-3.14.2.jar:?]
	at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.java:263) ~[okhttp-3.14.2.jar:?]
	... 28 more

With this PR it looks like the following. (multiple messages)

2019-07-31 23:58:08,707 WARN : org.graylog2.shared.rest.resources.ProxiedResource - Unable to call http://127.0.0.1:9010/api/system/metrics/multiple on node <38fe132d-0a9a-4194-ae18-73d9fd1e134c>: Failed to connect to /127.0.0.1:9010
2019-07-31 23:58:09,021 WARN : org.graylog2.shared.rest.resources.ProxiedResource - Unable to call http://127.0.0.1:9010/api/system/metrics/multiple on node <38fe132d-0a9a-4194-ae18-73d9fd1e134c>: Failed to connect to /127.0.0.1:9010
2019-07-31 23:58:09,479 WARN : org.graylog2.shared.rest.resources.ProxiedResource - Unable to call http://127.0.0.1:9010/api/system/inputstates on node <38fe132d-0a9a-4194-ae18-73d9fd1e134c>: Failed to connect to /127.0.0.1:9010
2019-07-31 23:58:10,471 WARN : org.graylog2.shared.rest.resources.ProxiedResource - Unable to call http://127.0.0.1:9010/api/system/metrics/multiple on node <38fe132d-0a9a-4194-ae18-73d9fd1e134c>: Failed to connect to /127.0.0.1:9010
2019-07-31 23:58:18,468 WARN : org.graylog2.shared.rest.resources.ProxiedResource - Unable to call http://127.0.0.1:9010/api/system/metrics/multiple on node <38fe132d-0a9a-4194-ae18-73d9fd1e134c>: Failed to connect to /127.0.0.1:9010
2019-07-31 23:58:19,016 WARN : org.graylog2.shared.rest.resources.ProxiedResource - Unable to call http://127.0.0.1:9010/api/system/metrics/multiple on node <38fe132d-0a9a-4194-ae18-73d9fd1e134c>: Failed to connect to /127.0.0.1:9010
Avoid logging excessive exceptions in ProxiedResource
Connection errors in ProxiedResource can happen if other nodes in the
cluster are restarting. Only log the stack trace if DEBUG is enabled for
the logger. Otherwise log a one-line message.

@bernd bernd added this to the 3.1.0 milestone Jul 31, 2019

@bernd bernd requested a review from mpfz0r Jul 31, 2019

@mpfz0r
mpfz0r approved these changes Aug 2, 2019

@mpfz0r mpfz0r merged commit 091cb9a into master Aug 2, 2019

3 of 4 checks passed

graylog-project/pr Jenkins build graylog-project-pr-snapshot 4996 has failed
Details
ci-web-linter Jenkins build graylog-pr-linter-check 4017 has succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
license/cla Contributor License Agreement is signed.
Details

@mpfz0r mpfz0r deleted the avoid-excessive-exceptions-proxied-resource branch Aug 2, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.