Skip to content
Browse files

add COPYRIGHT, GPLv3 licensing

  • Loading branch information...
hjmangalam committed Mar 21, 2019
1 parent a5a2083 commit cb1ae12dfd8f364230c16213b1f85fc73806f3b8
Showing with 90 additions and 21 deletions.
  1. +54 −0 COPYRIGHT
  2. +6 −3
  3. +30 −18 parsyncfp
@@ -0,0 +1,54 @@
...And The Small Print Taketh Away...

Copyright (c) 2019 by Harry Mangalam <>

Permission to use, copy, modify, and distribute this software and its
documentation is hereby granted, subject to the following restrictions
and understandings:

1) Any copy of this software or any copy of software derived
from it must include this copyright notice in full.

2) All materials or software developed as a consequence of the
use of this software or software derived from it must duly
acknowledge such use, in accordance with the usual standards
of acknowledging credit in academic research.

3) The software may be used by anyone for any purpose, subject to the
Free Software Foundation's General Public License v3, which can be viewed

Those persons or organizations wanting to use code derived from parsyncfp
in proprietary applications so as not to have their code 'infected'
by the FSF GPL have the option to license the code directly from from
the author under a separate license. This option takes advantage of
the so-called Dual Licensing policy, which is approved by the FSF.

4) This software is provided AS IS with NO WARRANTIES OF ANY KIND. The
author shall have no liability with respect to the infringement of
copyrights, trade secrets or any patents by this software or any part
thereof. In no event will the author be liable for any lost revenue
or profits or other special, indirect and consequential damages.

Inclusion of other software

This software uses or depends on code from other authors and packages that are
the copyright of the individual authors. Their release policies are not in
conflict with those of the GPL, as long as their code is not incorporated into
parsyncfp. The externally developed code is described below.

The use of the included code does not imply an endorsement of the parsyncfp
package by the individual authors.

This software uses
- rsync (Andrew Tridgell and Paul Mackerrasas), <>
- fpart (Ganael LAPLANCHE <>),
- perfquery < hal.rosenstock [at] >
- ibstat < hal.rosenstock [at] >
- and many standard Linux utilities, any one of which could change
output format at any time and hobble parsyncfp. These include:
ifconfig, ethtool, iwconfig, netstat, route, cat, ls, awk, grep,
ps, wc, sort, tr, etc.

@@ -1,5 +1,5 @@
# parsyncfp
a parallel rsync wrapper in Perl
a parallel rsync wrapper in Perl. Released under GPL v3.

(Version changes moved to the bottom of this file)

@@ -13,8 +13,9 @@ chunk by chunk basis. This allows pfp to begin transferring files before the
complete recursive descent of the source dir is complete. This feature can save many
hours of prep time on very large dir trees.

Dependencies: In addition to the above "fpart", pfp also requires 2 other Perl-based utilities, [scut]( or [via git]( and [stats]( or via the same git as scut.

parsyncfp requires 'perfquery' and 'ibstat', Infiniband utilities both written
by Hal Rosenstock < hal.rosenstock [at] > if your use involves transit
over IB networks.

pfp is primarily tested on Linux, but is being ported to MacOSX
as well.
@@ -200,6 +201,8 @@ The correct version of the above command is:
## Changes
### 1.57
- added explicit GPL v3 licence
### 1.56
- added a better measurement of TCP bytes sent (via /proc/net/dev)
@@ -1,4 +1,11 @@
#!/usr/bin/env perl
# this software is Copyright Harry Mangalam <> 2019 and on.
# parsyncfp is released under the GPLv3 License. <>
# see also: <>
# Please see the file "COPYRIGHT" that should accompany this program.
# If not it is available at the parsyncfp github site:

use strict;
use Getopt::Long; # for std option handling: -h --yadda=badda, etc
use Socket;
@@ -9,13 +16,17 @@ use Term::ANSIColor; # for alarms
# after significant changes, update the tarball and cp to moo for distribution; update the github
# fn="/home/hjm/bin/parsyncfp"; cd ; cp $fn ~/parsyncfp/; tar -cvzf parsyncfp+utils.tar.gz parsyncfp; scp parsyncfp+utils.tar.gz moo:~/public_html/parsync ;
# copy to all the local hosts
# scp $fn moo:~/public_html; scp $fn moo:~/bin; scp $fn dabrick:~/bin; ssh -t moo 'scp bin/parsyncfp hmangala@hpcs:/data/hpc/bin';

# don't forget!! using git! Add changes to changelog in the file
# cd ~/gits/parsyncfp; cp ~/bin/parsyncfp .; git add parsyncfp; git commit -m 'commit message'; git push
# check github for bug reports.

# - integrate pmj into pfp? use somthing like [xterm -e "cd /path/to/pmj/dir; pmj shell start file; wait"

# - --pmj=/path/to/pmj dir
# - starts an xterm and sends output there, opens the gnuplot window
# - [x] changed the calc for determining TCP network bandwidth to reference /proc/net/dev which should
# be more reliable across distro's and maybe even OSs. However, this won't detect RDMA data. For that,
# need perfquery.
@@ -30,7 +41,7 @@ use Term::ANSIColor; # for alarms
# - [x] done: fix bandwidth calculation subroutine.
# - [x] done: check for fpart before running.
# - [x] done: test for '-d' or --delete' in the rsyncopts line and refer to problem with this.
# - test for # of chunk files generated. emit warnings if goes above 2000 (admise to choose
# - test for # of chunk files generated. emit warnings if goes above 2000 (advise to choose
# a larger chunksize; or if less than NP. Don't assume a large # or even the same # as the NP #.
# - [x] done: decouple the cycle time from the job start time. ie, keep monitoring the exit codes
# and launch the next rsync immediately, don't wait for the checkperiod cycle, since that could be
@@ -53,8 +64,8 @@ $PERFQUERY $avgTCPrecv $avgTCPsend $avgRDMArecv $avgRDMAsend

parsyncfp version 1.56
Dec 12th, 2018
parsyncfp version 1.57
Mar 20, 2019
by Harry Mangalam <hjmangalam\>
parsyncfp is a Perl script that wraps Andrew Tridgell's miraculous
@@ -63,6 +74,7 @@ network connections to increase the amount of bandwidth it can use.
The 'fp' variant uses 'fpart' to bypass the need for a full recursive
descent of the dir trees before the actual transfer starts.
Do NOT try to use rsync --delete options'. More help with '--help'
parsyncfp is distributed under the Gnu Public License (GPL) v3.

if (! @ARGV) {usage();} # in case someone doesn't know what to do.
@@ -173,13 +185,13 @@ $NETIF \nPlease specify the one you want to use via the '--interface' flag.\n";
my $pqpath = `which perfquery`;
if ($NETIF =~ /ib/){
INFO("You've specified what looks like an Infiniband interface [$NETIF]...");
INFO("You've specified what looks like an Infiniband interface [$NETIF]...\n");
if ($pqpath ne "") {
INFO(".. and you have 'perfquery installed, so RDMA bytes will be reported as well.");
INFO(".. and you have 'perfquery installed, so RDMA bytes will be reported as well.\n");
} else {
INFO(".. but you don't have 'perfquery' installed, so only TCP bytes will be reported.");
INFO(".. but you don't have 'perfquery' installed, so only TCP bytes will be reported.\n");

@@ -236,7 +248,7 @@ if (-d $parsync_dir) {
elsif ($VERBOSE > 0) {pause();}
system("rm -f $glob");
if ($VERBOSE >=2 ) {
INFO("The fpart chunk files [$glob] are cleared .. continuing.")
INFO("The fpart chunk files [$glob] are cleared .. continuing.\n")
} elsif (! -d $parsync_dir) {
make_path $parsync_dir or FATAL("Can't create [ $parsync_dir ]");
@@ -309,15 +321,15 @@ $fnd2r =~ s/^\s+|\s+$//g ; # trim leading and trailing
my $cmd = "fpart -v -L -z -s $FPARTSIZE_N -o $FP_ROOT $fnd2r 2> $FPART_LOGFILE & echo \"\${!}\" > $FP_PIDFILE"; # captures the child PID!
if ($DEBUG) {print "DEBUG: fpart fork cmd:\n[$cmd]\n";} sleep 5;
if ($FPART_PID = fork) { # this actually takes a couple of seconds
if ($VERBOSE >= 2) { INFO("Forking fpart. Check [$FPART_LOGFILE] for errors if it hangs."); }
if ($VERBOSE >= 2) { INFO("Forking fpart. Check [$FPART_LOGFILE] for errors if it hangs.\n"); }
} else {
system "$cmd";
exit(0); # it's forked, now exit this stanza
# fpart has been forked; wait for enough chunkfiles to be written to start the rsyncs
while (! -e $FP_PIDFILE) { sleep 1;
if ($VERBOSE >= 3) {INFO("Waiting for fpart to be forked..");}
if ($VERBOSE >= 3) {INFO("Waiting for fpart to be forked..\n");}
my $ready2start = my $waitcnt = $NBR_FP_FLES = 0;
@@ -326,11 +338,11 @@ my $fp1 = $FP_ROOT . ".1";
my $done = 0;
while ($ready2start == 0) {
if (-e $fp0) {
if ($VERBOSE >= 3) {INFO("[$fp0] visible.");}
if ($VERBOSE >= 3) {INFO("[$fp0] visible.\n");}
$NBR_FP_FLES++; $ready2start=1;
if ($VERBOSE >= 3) {INFO("Waiting [$waitcnt]s for chunk files to be written");}
if ($VERBOSE >= 3) {INFO("Waiting [$waitcnt]s for chunk files to be written\r");}
sleep 1;

@@ -341,7 +353,7 @@ my $FPCFS = "${FP_ROOT}."; # FP Chunk File Stem
my $NBR_FP_FLES = `\\ls -U1 ${FPCFS}* | wc -l`; chomp $NBR_FP_FLES;
$RSYNCS_GOING = $CUR_FPI = 0; # $CUR_FPI = current FP index

if ($VERBOSE >= 2) {INFO("Starting the 1st [$NP] rsyncs ..");}
if ($VERBOSE >= 2) {INFO("Starting the 1st [$NP] rsyncs ..\n");}
my $sc = 0;
while ($RSYNCS_GOING < $NP && $KEEPGOING) { #
$CUR_FP_FLE = $FP_ROOT . "." . $CUR_FPI ; # the current fp chunkfile
@@ -562,7 +574,7 @@ Reminder: check the parsyncfp log [$logfile]
# OR keep cycling continuously on a 1s loop and ONLY print out info every X cycles. This is the easiest way.
if (-e $CUR_FP_FLE) {
fixfilenames($CUR_FP_FLE, $ROOTDIR); # check & fix for spaces, bad chars.
if ($VERBOSE >= 3) {my $tt = $CUR_FPI+1; INFO("next chunk [$tt] of [$nbr_cur_fpc_fles]");}
if ($VERBOSE >= 3) {my $tt = $CUR_FPI+1; INFO("next chunk [$tt] of [$nbr_cur_fpc_fles]\n");}
system("$RSYNC_CMD"); # capture the bg job PID to PIDfile
@@ -598,7 +610,7 @@ elsif ($DISPOSE =~ /c/ ) { # can it just be put into background?
Reminder: check [$FPART_LOGFILE] for errors if there were errors.
Thanks for using parsyncfp. Tell me how to make it better.

@@ -722,7 +734,7 @@ sub pause {
sub INFO($) {
my $msg = shift;
print color('bold blue');
print " INFO: $msg \n";
print " INFO: $msg";
print color('reset');

@@ -817,13 +829,13 @@ sub fix_ssh_config {
the run if you don't have ssh set correctly for the remote system, but the
transfer should still work.)
If this mod of your ~/.ssh/config file is OK, hit [Enter]. Otherwise hit [s] to skip. ");
If this mod of your ~/.ssh/config file is OK, hit [Enter]. Otherwise hit [s] to skip.\n ");
my $tmp = <STDIN>;
if ($tmp !~ /[sS]/) {
system ("echo -n \"#Next 2 lines added by parsyncfp\nForwardX11Trusted yes\nForwardX11 yes\n\" >> $HOME/.ssh/config" );
system("chmod 600 $HOME/.ssh/config");
INFO("Your ~/.ssh/config file is set correctly.\n"); sleep 1;
} else {INFO("Your ~/.ssh/config was not changed."); sleep 1;}
} else {INFO("Your ~/.ssh/config was not changed.\n"); sleep 1;}

0 comments on commit cb1ae12

Please sign in to comment.
You can’t perform that action at this time.