Sweeper is a small HDFS browser that let’s you visualize files stats on Hadoop. We use it at Ning to quickly find large directories to cleanup (with large amounts of small files) to keep the number of files managed by our namenode under control.
java -jar metrics.sweeper-*-jar-with-dependencies.jar
You can configure the namenode url via sweeper.hadoop.namenode.url:
java -Dsweeper.hadoop.namenode.url=hdfs://namenode.mycompany.com:9000 -jar metrics.sweeper-*-jar-with-dependencies.jar
You can configure the credentials via sweeper.hadoop.ugi (make sure to escape the coma):
java -Dsweeper.hadoop.ugi='hadoop-user\,hadoop-group' -jar metrics.sweeper-*-jar-with-dependencies.jar
By default, the browser will look at the entire cluster. You can specify a specific path via sweeper.hadoop.path:
java -Dsweeper.hadoop.path=/user/pierre -jar metrics.sweeper-*-jar-with-dependencies.jar
By default, files and directories size are shown (taking into account the replication factor). To show the number of files, use sweeper.content_summary=NUMBER_OF_FILES:
java -Dsweeper.content_summary=NUMBER_OF_FILES -jar metrics.sweeper-*-jar-with-dependencies.jar
mvn assembly:assembly
See COPYING file.