File utilities designed for scalability and performance.
Clone or download
dsikich docs: update dsync option for deleting destination files
    - update no-delete option to delete-dst option, which will
      delete extraneous files in the destination directory
Latest commit 3698654 Dec 13, 2018
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.github Add CONTRIBUTING.md Jul 10, 2017
auxdir remove more autotools cruft Jan 24, 2017
devops Use current vagrant box. Jan 14, 2014
doc docs: update dsync option for deleting destination files Dec 14, 2018
experimental promote dfind and dbz2 from experimental Sep 4, 2018
m4 no need for pandoc Apr 16, 2018
man man: fix automake to distribute man pages Aug 28, 2018
src dsync: do not delete files at the destination Dec 14, 2018
test Merge pull request #42 from hpc/sikich1/sync Jun 14, 2017
.gitignore ignore generated files Apr 16, 2018
.travis.yml update to dtcmp v1.1.0 Aug 21, 2018
AUTHORS Add Giuseppe Di Natale to AUTHORS Mar 28, 2017
LICENSE added all the names to license Aug 22, 2018
Makefile.am install man pages Aug 21, 2018
NOTICE clean up copyright and notice file Aug 22, 2018
README.md Remove redundant info from README Oct 17, 2018
astyle.options insert space in if,while,for and condition, drop break-blocks setting… May 21, 2014
autogen.sh Added autogen script. Dec 5, 2013
builddist update docs for v0.8 release Aug 21, 2018
buildme dropped too many files Aug 21, 2018
buildme_autotools updated build scripts for EA systems May 31, 2017
buildme_dependencies dropped too many files Aug 21, 2018
buildme_dependencies_dev buildme: add dev script to build dependencies needed for master Aug 22, 2018
buildme_dev dropping some old files Aug 21, 2018
buildrpm add build rpm support Mar 6, 2017
common.mk dropped too many files Aug 21, 2018
configure.ac promote dfind and dbz2 from experimental Sep 4, 2018
mpifileutils.spec update docs for v0.8 release Aug 21, 2018
mpifu_venv.sh fixing print error in venv script Jul 28, 2017

README.md

mpiFileUtils

mpiFileUtils provides both a library called libmfu and a suite of MPI-based tools to manage large datasets, which may vary from large directory trees to large files. High-performance computing users often generate large datasets with parallel applications that run with many processes (millions in some cases). However those users are then stuck with single-process tools like cp and rm to manage their datasets. This suite provides MPI-based tools to handle typical jobs like copy, remove, and compare for such datasets, providing speedups of up to 20-30x. It also provides a library that simplifies the creation of new tools or can be used in applications.

Documentation is available on ReadTheDocs.

Contributors

We welcome contributions to the project. For details on how to help, see our Contributor Guide

Copyrights

Copyright (c) 2013-2015, Lawrence Livermore National Security, LLC. Produced at the Lawrence Livermore National Laboratory CODE-673838

Copyright (c) 2006-2007,2011-2015, Los Alamos National Security, LLC. (LA-CC-06-077, LA-CC-10-066, LA-CC-14-046)

Copyright (2013-2015) UT-Battelle, LLC under Contract No. DE-AC05-00OR22725 with the Department of Energy.

Copyright (c) 2015, DataDirect Networks, Inc.

All rights reserved.

Build Status

The current status of the mpiFileUtils master branch is Build Status.