Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Don't log multi-megabyte guice exceptions. #13782

Closed
wants to merge 1 commit into from

Conversation

rmuir
Copy link
Contributor

@rmuir rmuir commented Sep 24, 2015

Instead just log the same thing we print to the startup console for that case (magic logic),
it sucks to do this, but guice exceptions are too much.

All other non-guice exceptions will still be fully logged.

cc: @drewr

Instead just log the same thing we print to the startup console for that case (magic logic),
it sucks to do this, but guice exceptions are too much.

All other non-guice exceptions will still be fully logged.
@drewr
Copy link
Contributor

drewr commented Sep 24, 2015

👍

@rmuir
Copy link
Contributor Author

rmuir commented Sep 24, 2015

For something like a bad network host, i get a 2KB log (the same magic logic going to console) instead of a 4MB one:

015-09-24 18:10:50,968][INFO ][node                     ] [Vapor] version[3.0.0-SNAPSHOT], pid[21417], build[a14a525/2015-09-24T22:08:16Z]
[2015-09-24 18:10:50,969][INFO ][node                     ] [Vapor] initializing ...
[2015-09-24 18:10:51,047][INFO ][plugins                  ] [Vapor] loaded [], sites []
[2015-09-24 18:10:51,060][INFO ][env                      ] [Vapor] using [1] data paths, mounts [[/ (/dev/disk/by-uuid/1b3a9e9c-3d5f-412e-be52-a7e64433e25d)]], net usable_space [154.1gb], net total_space [229.1gb], spins? [no], types [ext4]
[2015-09-24 18:10:52,151][ERROR][bootstrap                ] Guice Exception: java.lang.IllegalArgumentException: Failed to resolve address for [_bogus_]
Likely root cause: java.net.UnknownHostException: _bogus_: unknown error
        at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
        at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:907)
        at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1302)
        at java.net.InetAddress.getAllByName0(InetAddress.java:1255)
        at java.net.InetAddress.getAllByName(InetAddress.java:1171)
        at java.net.InetAddress.getAllByName(InetAddress.java:1105)
        at org.elasticsearch.transport.netty.NettyTransport.parse(NettyTransport.java:708)
        at org.elasticsearch.transport.netty.NettyTransport.addressesFromString(NettyTransport.java:660)
        at org.elasticsearch.transport.TransportService.addressesFromString(TransportService.java:399)
        at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing.<init>(UnicastZenPing.java:160)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
        at <<<guice>>>
        at org.elasticsearch.node.Node.<init>(Node.java:201)
        at org.elasticsearch.node.Node.<init>(Node.java:128)
        at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:143)
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:177)
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:277)
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)

I am the first to be against hiding exceptions in any way, but 4MB is over the top anywhere.

@rmuir
Copy link
Contributor Author

rmuir commented Sep 24, 2015

I'm setting this as a bug for 2.0, because I think it is too important. Feel free to uncheck that if you think it is wrong to do.

@drewr
Copy link
Contributor

drewr commented Sep 24, 2015

I think it's fairly critical as the experience for the operator wasn't great, and 2.0 is all about fixing that.

@rjernst
Copy link
Member

rjernst commented Sep 24, 2015

LGTM

@clintongormley clintongormley added the :Core/Infra/Logging Log management and logging utilities label Sep 25, 2015
@rmuir
Copy link
Contributor Author

rmuir commented Sep 25, 2015

Does anyone else have feelings/concerns on this? I know i do, about hiding exception stuff that may be important. That is why I always hate hiding any part of an exception/stacktrace in any way.

But 4MB is crazy, the way I see it, there has to be a line somewhere.

@jpountz
Copy link
Contributor

jpountz commented Sep 25, 2015

+1

@mikemccand
Copy link
Contributor

LGTM

@rmuir rmuir closed this in aa8d174 Sep 25, 2015
rmuir added a commit that referenced this pull request Sep 25, 2015
Instead just log the same thing we print to the startup console for that case (magic logic),
it sucks to do this, but guice exceptions are too much.

All other non-guice exceptions will still be fully logged.

Closes #13782
rmuir added a commit that referenced this pull request Sep 25, 2015
Instead just log the same thing we print to the startup console for that case (magic logic),
it sucks to do this, but guice exceptions are too much.

All other non-guice exceptions will still be fully logged.

Closes #13782
@sherry-ger
Copy link

This still seems to be an issue in ES 2.0. It generated 40 GB of logs with messages like the following.

[2015-11-17 17:00:07,435][WARN ][org.elasticsearch        ] Exception cause unwrapping ran for 10 levels...
RemoteTransportException[[mon-esd-2][10.1.108.103:9300][indices:data/write/bulk[s]]]; nested:
RemoteTransportException[[mon-esd-3][10.1.108.111:9300][indices:data/write/bulk[s]]]; nested: 
RemoteTransportException[[mon-esd-2][10.1.108.103:9300][indices:data/write/bulk[s]]]; nested: 
RemoteTransportException[[mon-esd-3][10.1.108.111:9300][indices:data/write/bulk[s]]]; nested: 
RemoteTransportException[[mon-esd-2][10.1.108.103:9300][indices:data/write/bulk[s]]]; nested: 
RemoteTransportException[[mon-esd-3][10.1.108.111:9300][indices:data/write/bulk[s]]]; nested: 
RemoteTransportException[[mon-esd-2][10.1.108.103:9300][indices:data/write/bulk[s]]]; nested: 
RemoteTransportException[[mon-esd-3][10.1.108.111:9300][indices:data/write/bulk[s]]]; nested: 
RemoteTransportException[[mon-esd-2][10.1.108.103:9300][indices:data/write/bulk[s]]]; nested: 
RemoteTransportException[[mon-esd-3][10.1.108.111:9300][indices:data/write/bulk[s]]]; nested: 
RemoteTransportException[[mon-esd-2][10.1.108.103:9300][indices:data/write/bulk[s]]]; nested: 
RemoteTransportException[[mon-esd-3][10.1.108.111:9300][indices:data/write/bulk[s]]]; nested: 
RemoteTransportException[[mon-esd-2][10.1.108.103:9300][indices:data/write/bulk[s]]]; nested: 
 ...
RemoteTransportException[[mon-esd-3][10.1.108.111:9300][indices:data/write/bulk[s]]]; nested: 
RemoteTransportException[[mon-esd-2][10.1.108.103:9300][indices:data/write/bulk[s]]]; nested: 
RemoteTransportException[[mon-esd-3][10.1.108.111:9300][indices:data/write/bulk[s]]]; nested: NullPointerException;
Caused by: RemoteTransportException[[mon-esd-2][10.1.108.103:9300][indices:data/write/bulk[s]]]; nested: 
RemoteTransportException[[mon-esd-3][10.1.108.111:9300][indices:data/write/bulk[s]]]; nested: 
RemoteTransportException[[mon-esd-2][10.1.108.103:9300][indices:data/write/bulk[s]]]; nested: 
RemoteTransportException[[mon-esd-3][10.1.108.111:9300][indices:data/write/bulk[s]]]; nested: 

@rmuir
Copy link
Contributor Author

rmuir commented Nov 19, 2015

That is because this is not a fix. The fix is to remove guice.

@bleskes
Copy link
Contributor

bleskes commented Nov 19, 2015

@sherry-ger I wonder whether this is guice driven - do you have the bottom of the stack trace / something else? do you know where is that NullPointerException coming from?

@sherry-ger
Copy link

@bleskes No, this probably is not guice driven. This is more akin to #4639, which seems to suggest this feature may have resolved it. I do have the rest of the stack trace. I will post this to the right issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
>bug :Core/Infra/Logging Log management and logging utilities v2.1.0 v5.0.0-alpha1
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants