Permalink
Browse files

HADOOP-7154. Set MALLOC_ARENA_MAX in hadoop-env.sh. Contributed by To…

…dd Lipcon.

git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0@1364929 13f79535-47bb-0310-9956-ffa450edef68
  • Loading branch information...
mattf-horton committed Jul 24, 2012
1 parent 373f1e6 commit 4376422f4303338fc5b27b840488b7c86b00f315
Showing with 8 additions and 0 deletions.
  1. +3 −0 CHANGES.txt
  2. +5 −0 bin/hadoop-config.sh
View
@@ -6,6 +6,9 @@ Release 1.0.4 - Unreleased
IMPROVEMENTS
+ HADOOP-7154. Should set MALLOC_ARENA_MAX in hadoop-env.sh
+ (todd via mattf)
+
BUG FIXES
HDFS-3652. FSEditLog failure removes the wrong edit stream when storage
View
@@ -75,6 +75,11 @@ if [ "$HADOOP_HOME_WARN_SUPPRESS" = "" ] && [ "$HADOOP_HOME" != "" ]; then
echo 1>&2
fi
+# Newer versions of glibc use an arena memory allocator that causes virtual
+# memory usage to explode. This interacts badly with the many threads that
+# we use in Hadoop. Tune the variable down to prevent vmem explosion.
+export MALLOC_ARENA_MAX=${MALLOC_ARENA_MAX:-4}
+
export HADOOP_HOME=${HADOOP_PREFIX}
export HADOOP_HOME_WARN_SUPPRESS=1

0 comments on commit 4376422

Please sign in to comment.