[STORM-1215] Use Asynchronous logger for improving performance.

[kishorvpatil]

The changes to loggers for all daemons and workers.

• Turn off immediateFlush.
• Enable async logging.
• Turn off Immediate Fail for log events on syslog logger to avoid making blocking call without timeout.

[d2r]

This also switches to UTF-8 encoding for syslog, for compatibility. We have seen issues with this in the past.

Evidently we cannot configure the context except via a System property, so looks OK to me.

+1

[revans2]

The log level changes on the UI still work, and performance seems fine too, but if I enable topology.debug I am seeing some very odd things that I would like to explore some more.

[revans2]

OK I got a heap dump and when debug is enabled we are logging faster then the logger can write. This is resulting in the async logger disruptor queue starting to fill up, and making lots of GC problems. Can we disable this for the worker process or find a way to configure it so the memory usage is not unbounded?

[kishorvpatil]

@revans2 Thank you for testing and quick feedback on it. I have reverted the changes for worker, so worker will continue to log synchronously.

[revans2]

+1 I just tested and the workers are no longer crashing, but it might be good to set immediateFlush="false" in the worker.

[kishorvpatil]

@revans2 Turned off immediateFlush for worker.

[revans2]

+1 looks good to me

[revans2]

@d2r any other comments on the most recent changes?

[d2r]

Looks good to me +1

[rajeshksv]

You mean Turn ON Immediate Fail right ?