Hadoop utility to quickly find large directories to clean up or small files to combine.
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Failed to load latest commit information.



Sweeper is a small HDFS browser that let's you visualize files stats on Hadoop. We use it at Ning to quickly find large directories to cleanup (with large amounts of small files) to keep the number of files managed by our namenode under control.


java -jar metrics.sweeper-*-jar-with-dependencies.jar

You can configure the namenode url via sweeper.hadoop.namenode.url:

java  -Dsweeper.hadoop.namenode.url=hdfs://namenode.mycompany.com:9000 -jar metrics.sweeper-*-jar-with-dependencies.jar

You can configure the credentials via sweeper.hadoop.ugi (make sure to escape the coma):

java -Dsweeper.hadoop.ugi='hadoop-user\,hadoop-group' -jar metrics.sweeper-*-jar-with-dependencies.jar

By default, the browser will look at the entire cluster. You can specify a specific path via sweeper.hadoop.path:

java -Dsweeper.hadoop.path=/user/pierre -jar metrics.sweeper-*-jar-with-dependencies.jar

By default, files and directories size are shown (taking into account the replication factor). To show the number of files, use sweeper.content_summary=NUMBER_OF_FILES:

java -Dsweeper.content_summary=NUMBER_OF_FILES -jar metrics.sweeper-*-jar-with-dependencies.jar


mvn install

License (see COPYING file for full license)

Copyright 2010-2012 Ning

Licensed under the Apache License, Version 2.0 (the “License”); you may not use this file except in compliance with the License. You may obtain a copy of the License at


Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.