Don't log multi-megabyte guice exceptions.

[rmuir]

Instead just log the same thing we print to the startup console for that case (magic logic),
it sucks to do this, but guice exceptions are too much.

All other non-guice exceptions will still be fully logged.

cc: @drewr

[drewr]

👍

[rmuir]

For something like a bad network host, i get a 2KB log (the same magic logic going to console) instead of a 4MB one:

015-09-24 18:10:50,968][INFO ][node                     ] [Vapor] version[3.0.0-SNAPSHOT], pid[21417], build[a14a525/2015-09-24T22:08:16Z]
[2015-09-24 18:10:50,969][INFO ][node                     ] [Vapor] initializing ...
[2015-09-24 18:10:51,047][INFO ][plugins                  ] [Vapor] loaded [], sites []
[2015-09-24 18:10:51,060][INFO ][env                      ] [Vapor] using [1] data paths, mounts [[/ (/dev/disk/by-uuid/1b3a9e9c-3d5f-412e-be52-a7e64433e25d)]], net usable_space [154.1gb], net total_space [229.1gb], spins? [no], types [ext4]
[2015-09-24 18:10:52,151][ERROR][bootstrap                ] Guice Exception: java.lang.IllegalArgumentException: Failed to resolve address for [_bogus_]
Likely root cause: java.net.UnknownHostException: _bogus_: unknown error
        at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
        at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:907)
        at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1302)
        at java.net.InetAddress.getAllByName0(InetAddress.java:1255)
        at java.net.InetAddress.getAllByName(InetAddress.java:1171)
        at java.net.InetAddress.getAllByName(InetAddress.java:1105)
        at org.elasticsearch.transport.netty.NettyTransport.parse(NettyTransport.java:708)
        at org.elasticsearch.transport.netty.NettyTransport.addressesFromString(NettyTransport.java:660)
        at org.elasticsearch.transport.TransportService.addressesFromString(TransportService.java:399)
        at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing.<init>(UnicastZenPing.java:160)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
        at <<<guice>>>
        at org.elasticsearch.node.Node.<init>(Node.java:201)
        at org.elasticsearch.node.Node.<init>(Node.java:128)
        at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:143)
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:177)
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:277)
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)

I am the first to be against hiding exceptions in any way, but 4MB is over the top anywhere.

[rmuir]

I'm setting this as a bug for 2.0, because I think it is too important. Feel free to uncheck that if you think it is wrong to do.

[drewr]

I think it's fairly critical as the experience for the operator wasn't great, and 2.0 is all about fixing that.

[rjernst]

LGTM

[rmuir]

Does anyone else have feelings/concerns on this? I know i do, about hiding exception stuff that may be important. That is why I always hate hiding any part of an exception/stacktrace in any way.

But 4MB is crazy, the way I see it, there has to be a line somewhere.

[jpountz]

+1

[mikemccand]

LGTM

[sherry-ger]

This still seems to be an issue in ES 2.0. It generated 40 GB of logs with messages like the following.

[2015-11-17 17:00:07,435][WARN ][org.elasticsearch        ] Exception cause unwrapping ran for 10 levels...
RemoteTransportException[[mon-esd-2][10.1.108.103:9300][indices:data/write/bulk[s]]]; nested:
RemoteTransportException[[mon-esd-3][10.1.108.111:9300][indices:data/write/bulk[s]]]; nested: 
RemoteTransportException[[mon-esd-2][10.1.108.103:9300][indices:data/write/bulk[s]]]; nested: 
RemoteTransportException[[mon-esd-3][10.1.108.111:9300][indices:data/write/bulk[s]]]; nested: 
RemoteTransportException[[mon-esd-2][10.1.108.103:9300][indices:data/write/bulk[s]]]; nested: 
RemoteTransportException[[mon-esd-3][10.1.108.111:9300][indices:data/write/bulk[s]]]; nested: 
RemoteTransportException[[mon-esd-2][10.1.108.103:9300][indices:data/write/bulk[s]]]; nested: 
RemoteTransportException[[mon-esd-3][10.1.108.111:9300][indices:data/write/bulk[s]]]; nested: 
RemoteTransportException[[mon-esd-2][10.1.108.103:9300][indices:data/write/bulk[s]]]; nested: 
RemoteTransportException[[mon-esd-3][10.1.108.111:9300][indices:data/write/bulk[s]]]; nested: 
RemoteTransportException[[mon-esd-2][10.1.108.103:9300][indices:data/write/bulk[s]]]; nested: 
RemoteTransportException[[mon-esd-3][10.1.108.111:9300][indices:data/write/bulk[s]]]; nested: 
RemoteTransportException[[mon-esd-2][10.1.108.103:9300][indices:data/write/bulk[s]]]; nested: 
 ...
RemoteTransportException[[mon-esd-3][10.1.108.111:9300][indices:data/write/bulk[s]]]; nested: 
RemoteTransportException[[mon-esd-2][10.1.108.103:9300][indices:data/write/bulk[s]]]; nested: 
RemoteTransportException[[mon-esd-3][10.1.108.111:9300][indices:data/write/bulk[s]]]; nested: NullPointerException;
Caused by: RemoteTransportException[[mon-esd-2][10.1.108.103:9300][indices:data/write/bulk[s]]]; nested: 
RemoteTransportException[[mon-esd-3][10.1.108.111:9300][indices:data/write/bulk[s]]]; nested: 
RemoteTransportException[[mon-esd-2][10.1.108.103:9300][indices:data/write/bulk[s]]]; nested: 
RemoteTransportException[[mon-esd-3][10.1.108.111:9300][indices:data/write/bulk[s]]]; nested: 

[rmuir]

That is because this is not a fix. The fix is to remove guice.

[bleskes]

@sherry-ger I wonder whether this is guice driven - do you have the bottom of the stack trace / something else? do you know where is that NullPointerException coming from?

[sherry-ger]

@bleskes No, this probably is not guice driven. This is more akin to #4639, which seems to suggest this feature may have resolved it. I do have the rest of the stack trace. I will post this to the right issue.