Add way to know when rsync is complete

[shunt16]

Hi @corizzi - it would be really helpful in the hypernets processor to be able to tell when the data has finished transferring to the server, so we know when to begin processing. We originally opened an issue for this in hypernets_processor, where @jkuusk made a suggestion for a fix. Thanks!

[corizzi]

Hi @shunt16 ,

As LOGS/ is synchronized after DATA/:

hypernets_tools/utils/hello_server.sh

Lines 88 to 91 in 93984a4

	echo "Syncing Data..."
	rsync -e "ssh -p $sshPort" -rt --exclude "CUR*" "DATA" "$ipServer:$remoteDir"
	echo "Syncing Logs..."
	rsync -e "ssh -p $sshPort" -rt "LOGS" "$ipServer:$remoteDir"

We can also use the respective timestamps of each folder, for example:

dataTimeStamp=$(stat -c %Y DATA/)
logsTimeStamp=$(stat -c %Y LOGS/)

if [ "$dataTimeStamp" -lt "$logsTimeStamp" ] ; then
    echo "Sync is done!"
else
    echo "Syncing..."
fi

[pdevis]

OK we can use the log files to check if data has finished syncing then. Thanks

[shunt16]

Hi @corizzi - I'm a little confused by the naming of the log files, and was wondering if you could help. How do you know which log file is related to which sequence? The name of the log file seems to be somehow offset from the related sequence. Would it be possible for the log file to include the name of the sequence?

[corizzi]

Hi @shunt16 ,

I added a line at the end of the log that must show something like:
Created sequence : DATA/SEQYYYYDDMMTHHMMSS

The naming of the log files are somehow offset because the date-time creation of the log is used. It is a bit confusing but a convenient choice as a "boot" does not necessary contain a sequence creation.

In any case, if you are looking for the sequence folder name according to a specific log file: yyyy-mm-dd-sequence.log
you can try to use the following bash code snippet:

f=yyyy-mm-dd-sequence.log
sequenceName=$(grep -m1 "CUR" $f)
sequenceName=$(echo "$sequenceName" | cut -d'/' -f2)
sequenceName=$(echo "SEQ${sequenceName:3}")

I hope it helps