Home

This documentation just show how can use added features.

See Hadoop Documentation for general use.

---

Balanced DataNode's local disk use ratio in several different disk capacity

When a file is written to HDFS, it is split up into blocks. They is placed a random node.(In this case we don't consider Rack Awareness.) For this reason, DataNodes' local disk used capacity is almost equally.

However, when our cluster's DataNodes have different disk capacity such as 500GB, 1TB and 4TB, firstly a smaller disk will be full. Typically a cluster administrator receives disk capacity alarm to avoid disk full issue. So even though larger disk has enough capacity, the administrator will receive a lot of disk capacity alarms because a smaller disk will be full.

We add new block placement policy: dfs.namenode.replication.considerDfsUsedPercent. It means when the NameNode chooses a DataNode for block placement, it consider disk's used ratio such as 10% or 30%. The configurations is like this:

 <property>
   <name>dfs.namenode.replication.considerDfsUsedPercent</name>
   <value>true</value>
   <description>Whether to enable considering dfs used percent</description>
 </property>
 <property>
   <name>dfs.namenode.replication.considerDfsUsedPercent.factor</name>
   <value>1.1</value>
   <description>If DataNode's disk used ratio is over (threshold.percent * factor), the Namenode doesn't consider used ratio, so the block placement is random.</description>
 </property>
 <property>
   <name>dfs.namenode.replication.considerDfsUsedPercent.threshold.percent</name>
   <value>60</value>
   <description>If DataNode's disk used ratio is less than this value, the Namenode doesn't consider used ratio,
 so the block placement is random.</description>
 </property>

Choose native library considering node' OS automatically

In YARN, applications perform their work by running containers and containers have dependencies on files for execution. At the time of starting a container, an ApplicationMaster can specify all the files that a container will require and thus should be localized. Localization is the process of copying/download remote resources onto the local file-system. The local file-system's resources(are called LocalResource) represents a file/library required to run a container.

The native library in LocalResource, such as .so, depends on node's OS. For example if you build a library on CentOS, maybe it doesn't work on Ubuntu. If you build it on CentOS 7, maybe it doesn't work on CentOS 6. If all of your nodes don't have the same OS, sometimes a job will be failed. Because there is a difference between running OS and building OS.

We improve localization to choose a library considering node's OS. The usage is like this:

1. You build libraries for each OS type to use. For example you have the nodes using CentOS 6 and 7, you build two libraries.
2. You add pre-defined prefix(_native_<os type>_) to file name. If the library is foo.so and you build a library for CentOS 6, the library will be _native_centos6_foo.so. Currently we support two pre-defined prefix: _native_centos6_ and _native_centos7_ .
3. You set the DistributedCache configuration to use the libraries.

   mapreduce.job.cache.files=/file/_native_centos6_foo.bar,/file/_native_centos7_foo.bar
   

This feature is also available for non-mapreduce frameworks such as Apache Spark.