YARN is killing containers

[keith-turner]

I have noticed that while running webindex for long periods on EC2 that YARN kills Fluo workers because they are using too much memory. I think if the java -Xmx setting may be too close to the Yarn memory limit. Need to figure out how to adjust this using Twill.

[keith-turner]

Used the advice in this email from twill dev list and it worked really well. Set twill to reserve 1G and gave workers 5G while running webindex and workers ran for long periods of time w/o being killed.

[keith-turner]

I added support for configuring the reserved memory in apache/fluo-muchos#139 and its working well.

[keith-turner]

Re-opened this because I realized Fluo core needs some documentation for this issue.