dcp is a distributed file copy program that automatically distributes and dynamically balances work equally across nodes in a large distributed system without centralized state.
Shell C
Permalink
Failed to load latest commit information.
auxdir Basic autotools checks for MPI and XATTRs. May 5, 2013
doc Bumped version to 0.1.0-rc.2. The minor number is changed due to feature Jul 3, 2013
m4 Added m4 for MPI and XATTR checks. May 5, 2013
src fix types to avoid compiler warnings Aug 14, 2013
tests Bumped version number. Modified tests to use in-tree tmp files. Jan 8, 2013
.gitignore Bumped version number. Modified tests to use in-tree tmp files. Jan 8, 2013
.travis.yml Addition of libattr1-dev to travis-ci config. May 5, 2013
AUTHORS Added Adam to authors list. Alphabetized. May 5, 2013
COPYING Updated date in copying file. Mar 9, 2012
HACKING.md Added note on fault tolerance. Jun 20, 2013
Makefile.am Addition of make test target. Dec 14, 2012
Makefile.in Basic autotools checks for MPI and XATTRs. May 5, 2013
README.md Removed unnecessary and untested file size limits note. Nov 12, 2013
aclocal.m4 Basic autotools checks for MPI and XATTRs. May 5, 2013
astyle.options Common style. Mar 1, 2012
autogen.sh Initial commit. Pulled from old libcircle examples and put into a pro… Oct 6, 2011
common.mk Removed overflow traps since they were causing clang linking warnings. Jan 9, 2013
config.h.in Basic autotools checks for MPI and XATTRs. May 5, 2013
configure Bumped version to 0.1.0-rc.2. The minor number is changed due to feature Jul 3, 2013
configure.ac Bumped version to 0.1.0-rc.2. The minor number is changed due to feature Jul 3, 2013
dcp.spec Bumped version to 0.1.0-rc.2. The minor number is changed due to feature Jul 3, 2013

README.md

dcp

A tool to copy file(s) in parallel on a distributed system.

SYNOPSIS

dcp [cCdfhpRrUv] [--] source_file target_file
dcp [cCdfhpRrUv] [--] source_file ... target_directory

DESCRIPTION

dcp is a file copy tool in the spirit of cp(1) that evenly distributes work across a large cluster without any centralized state. It is designed for copying files which are located on a distributed parallel file system. The method used in the file copy process is a self-stabilization algorithm which enables per-node autonomous processing and a token passing scheme to detect termination.

PREREQUISITES

An MPI environment is required (such as Open MPI's mpirun(1)) as well as the self-stabilization library known as LibCircle.

OPTIONS

-c, --conditional

When copying a source directory to a destination directory, copy the source directory over the destination directory. The default behavior is to copy the source directory inside the destination directory.

-C, --skip-compare

Skip the compare operation to confirm file integrity. When using this option, a file integrity check, such as md5sum, should be performed after the file(s) have been copied.

-d , --debug=level

Specify the level of debug information to output. Level may be one of: fatal, err, warn, info, or dbg. Increasingly verbose debug levels include the output of less verbose debug levels.

-f, --force

Remove existing destination files if creation or truncation fails. If the destination filesystem is specified to be unreliable (-U, --unreliable-filesystem), this option may lower performance since each failure will cause the entire file to be invalidated and copied again.

-h, --help

Print a brief message listing the dcp(1) options and usage.

-p, --preserve

Preserve the original files' owner, group, permissions (including the setuid and setgid bits), time of last modification and time of last access. In case duplication of owner or group fails, the setuid and setgid bits are cleared.

-R, --recursive

Copy directories recursively, and do the right thing when objects other than ordinary files or directories are encountered.

-r, --recursive-unspecified

Copy directories recursively, and ignore objects other than ordinary files or directories.

-U, --unreliable-filesystem

If the filesystem is very unreliable, this option may be used to always retry an operation when a failure occurs. If failures are permanent, this option will cause an infinite loop. Specifying this option when force is enabled (-f, --force) may lower performance.

-v, --version

Print version information and exit.

Known bugs

When the force option is specified and truncation fails, the copy and truncation will be stuck in an infinite loop until the truncation operation returns with success.

The maximum supported filename length for any file transfered is approximately 4068 characters. This may be less than the number of characters that your operating system supports.

RPM Creation

First, check the Build Status. If all tests are passing, create an rpm using the following instructions:

  1. rpmbuild -ta dcp-<version>.tar.gz
  2. rpm --install <the appropriate RPM files>

Contributions

Please view the HACKING.md file for more information on how to contribute to dcp.

COPYING

See the included COPYING file for additional information.