Skip to content

Commit

Permalink
Merge pull request #117 from gonsie/rtfd2
Browse files Browse the repository at this point in the history
Rtfd2
  • Loading branch information
dsikich committed Apr 18, 2018
2 parents a5fc72c + 2ebc5b8 commit 4c3550c
Show file tree
Hide file tree
Showing 7 changed files with 32 additions and 18 deletions.
25 changes: 13 additions & 12 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,24 +1,25 @@
# mpiFileUtils
mpiFileUtils is a suite of MPI-based tools to manage large datasets, which may vary from large directory trees to large files. High-performance computing users often generate large datasets with parallel applications that run with many processes (millions in some cases). However those users are then stuck with single-process tools like cp and rm to manage their datasets. This suite provides MPI-based tools to handle typical jobs like copy, remove, and compare for such datasets, providing speedups of up to 20-30x.
Documentation is available on [ReadTheDocs](http://mpifileutils.readthedocs.io) and in this repo:

- [dbcast](doc/markdown/dbcast.1.md) - Broadcast files to compute nodes.
- [dchmod](doc/markdown/dchmod.1.md) - Change permissions and group access on files.
- [dcmp](doc/markdown/dcmp.1.md) - Compare files.
- [dcp](doc/markdown/dcp.1.md) - Copy files.
- [dfilemaker](doc/markdown/dfilemaker.1.md) - Generate random files.
- [drm](doc/markdown/drm.1.md) - Remove files.
- [dstripe](doc/markdown/dstripe.1.md) - Restripe files.
- [dwalk](doc/markdown/dwalk.1.md) - List files.
- [dbcast](doc/rst/dbcast.1.rst) - Broadcast files to compute nodes.
- [dchmod](doc/rst/dchmod.1.rst) - Change permissions and group access on files.
- [dcmp](doc/rst/dcmp.1.rst) - Compare files.
- [dcp](doc/rst/dcp.1.rst) - Copy files.
- [dfilemaker](doc/rst/dfilemaker.1.rst) - Generate random files.
- [drm](doc/rst/drm.1.rst) - Remove files.
- [dstripe](doc/rst/dstripe.1.rst) - Restripe files.
- [dwalk](doc/rst/dwalk.1.rst) - List files.

## Experimental Utilities
To enable experimental utilities, run configure with the enable experimental option.

./configure --enable-experimental

- [dfind](doc/markdown/dfind.1.md) - Find files by path name (experimental).
- [dgrep](doc/markdown/dgrep.1.md) - Search contents of files (experimental).
- [dparallel](doc/markdown/dparallel.1.md) - Perform commands in parallel (experimental).
- [dtar](doc/markdown/dtar.1.md) - Create file tape archives (experimental).
- [dfind](doc/rst/dfind.1.rst) - Find files by path name (experimental).
- [dgrep](doc/rst/dgrep.1.rst) - Search contents of files (experimental).
- [dparallel](doc/rst/dparallel.1.rst) - Perform commands in parallel (experimental).
- [dtar](doc/rst/dtar.1.rst) - Create file tape archives (experimental).

## Build
mpiFileUtils depends on several libraries. To simplify builds, there are two scripts: buildme\_dependencies and buildme. The buildme\_dependencies script downloads and installs all the necessary libraries. The buildme script then builds mpiFileUtils assuming the libraries have been installed. Both scripts require that mpicc is in your path, and that it is for an MPI library that supports at least v2.2 of the MPI standard. Please review each buildme script, and edit if necessary. Then run them in sequence:
Expand Down
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
12 changes: 12 additions & 0 deletions doc/rst/experimental/index.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
.. mpiFileUtils documentation experimental index file.
Experimental Tools
==================

.. toctree::
:maxdepth: 1

dfind.1
dgrep.1
dparallel.1
dtar.1
13 changes: 7 additions & 6 deletions doc/rst/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -10,18 +10,19 @@ Documentation for mpiFileUtils
:maxdepth: 1

dbcast.1
dchmod.1
dcmp.1
dcp.1
dfilemaker.1
dgrep.1
drm.1
dtar.1
dchmod.1
dcp.1
dfind.1
dparallel.1
dstripe.1
dwalk.1

.. toctree::
:maxdepth: 2

experimental/index


Indices and tables
==================
Expand Down

0 comments on commit 4c3550c

Please sign in to comment.