Join GitHub today
GitHub is home to over 20 million developers working together to host and review code, manage projects, and build software together.
Tracking node memory usage in job scheduler #300
Although we use ulimit to limit an individual tasks memory usage, it is possible for several large tasks to consume all the memory on a node, without consuming all the slots on the node. If we don't leave some memory headroom on the node, strange failures occur, both in tasks as well as in disco/ddfs.