File utilities designed for scalability and performance.
Clone or download
dsikich Merge pull request #201 from hpc/dcmp-lite
Fix dcmp lite mode metadata check
Latest commit 99b5880 Nov 28, 2018
Type Name Latest commit message Commit time
Failed to load latest commit information.
.github Add Jul 10, 2017
auxdir remove more autotools cruft Jan 24, 2017
devops Use current vagrant box. Jan 14, 2014
doc dsync: add --batch-files option Nov 7, 2018
experimental promote dfind and dbz2 from experimental Sep 4, 2018
m4 no need for pandoc Apr 16, 2018
man man: fix automake to distribute man pages Aug 28, 2018
src Merge pull request #201 from hpc/dcmp-lite Nov 28, 2018
test Merge pull request #42 from hpc/sikich1/sync Jun 14, 2017
.gitignore ignore generated files Apr 16, 2018
.travis.yml update to dtcmp v1.1.0 Aug 21, 2018
AUTHORS Add Giuseppe Di Natale to AUTHORS Mar 28, 2017
LICENSE added all the names to license Aug 22, 2018 install man pages Aug 21, 2018
NOTICE clean up copyright and notice file Aug 22, 2018 Remove redundant info from README Oct 17, 2018
astyle.options insert space in if,while,for and condition, drop break-blocks setting… May 21, 2014 Added autogen script. Dec 5, 2013
builddist update docs for v0.8 release Aug 21, 2018
buildme dropped too many files Aug 21, 2018
buildme_autotools updated build scripts for EA systems May 31, 2017
buildme_dependencies dropped too many files Aug 21, 2018
buildme_dependencies_dev buildme: add dev script to build dependencies needed for master Aug 22, 2018
buildme_dev dropping some old files Aug 21, 2018
buildrpm add build rpm support Mar 6, 2017 dropped too many files Aug 21, 2018 promote dfind and dbz2 from experimental Sep 4, 2018
mpifileutils.spec update docs for v0.8 release Aug 21, 2018 fixing print error in venv script Jul 28, 2017


mpiFileUtils provides both a library called libmfu and a suite of MPI-based tools to manage large datasets, which may vary from large directory trees to large files. High-performance computing users often generate large datasets with parallel applications that run with many processes (millions in some cases). However those users are then stuck with single-process tools like cp and rm to manage their datasets. This suite provides MPI-based tools to handle typical jobs like copy, remove, and compare for such datasets, providing speedups of up to 20-30x. It also provides a library that simplifies the creation of new tools or can be used in applications.

Documentation is available on ReadTheDocs.


We welcome contributions to the project. For details on how to help, see our Contributor Guide


Copyright (c) 2013-2015, Lawrence Livermore National Security, LLC. Produced at the Lawrence Livermore National Laboratory CODE-673838

Copyright (c) 2006-2007,2011-2015, Los Alamos National Security, LLC. (LA-CC-06-077, LA-CC-10-066, LA-CC-14-046)

Copyright (2013-2015) UT-Battelle, LLC under Contract No. DE-AC05-00OR22725 with the Department of Energy.

Copyright (c) 2015, DataDirect Networks, Inc.

All rights reserved.

Build Status

The current status of the mpiFileUtils master branch is Build Status.