[xrdcp] option to continue after errors?

[cshimmin]

Hi,
I'm using xrdcp client v4.2.3. Is it possible to continue running in the case of errors (specifically, when running in --recursive mode)? The two scenarios where I find this would be useful are:

Corrupted data

You have 1000+ large files in some directory, that you'd like to copy. So you spin up xrdcp in a tmux session and let it go overnight.
If there is an error for some reason (e.g. [ERROR] Received corrupted data) on say, file number 5, you're hosed (and not very happy in the morning). This is because currently xrdcp just calls it quits and doesn't even try to transfer the remaining 994 files.

Existing data

Sometimes I add new data to a directory somewhere, and want to sync that to xrootd, again using the --recursive option. Currently xrootd will quit at the first file that it encounters which already exists at the destination. It would be useful if you could ask it to simply skip these files, and only transfer the new ones.
This could even be a less general case of allowing errors, e.g. by adding a --skipexisting option.

[abh3]

No, we don't have such an option. I will tag this as a requested enhancement.

[cshimmin]

Note that I found a partial work-around; by using the --parallel N option, it works as desired. If files already exist (or if there's any error for that matter), the parallel job slot just advances to the next job.

However --parallel 1 seems to be detected by xrdcp as a special case, and is run the normal way (without job slots) and will halt upon any error. Hence it is only a partial work-around since sometimes it's not appropriate to use parallel jobs (i.e. when disk I/O bound).

[simonmichal]

Should this be an option or default behaviour ? Since currently the behaviour is bit inconsistent (with --parallel xrdcp tries to copy all the files, without aborts after first failure) it would be good to standardize ?

[simonmichal]

Fixed by 92bdf14