"rucio download --no-subdir" behavior is different from its documentation

[rcarpa]

Motivation

Rucio documentation for file download states that the "--no-subdir" will overwrite files if they exist in the destination folder.

  --no-subdir           Don't create a subdirectory for the scope of the
                        files. Existing files in the directory will be
                        overwritten.

However, the files are not overwitten, as shown by the following example :

$ echo "some_content" > /tmp/test/a_file
$ rucio upload --rse XRD1 --scope test /tmp/test/a_file
$ echo "other_content" > /tmp/test/a_file 
$ rucio download --no-subdir --dir /tmp/test/ test:a_file
   ----------------------------------
   Download summary
   ----------------------------------------
   DID mock:a_file
   Total files:                                  1
   Downloaded files:                             0
   Files already found locally:                  1
   Files that cannot be downloaded:              0
$ cat /tmp/test/a_file 
other_content
$ rm /tmp/test/a_file
$ rucio download --no-subdir --dir /tmp/test/ test:a_file
   ----------------------------------
   Download summary
   ----------------------------------------
   DID mock:a_file
   Total files:                                  1
   Downloaded files:                             1
   Files already found locally:                  0
   Files that cannot be downloaded:              0
# cat /tmp/test/a_file 
some_content

Modification

Either the documentation must be updated to note that existing files will not be overwritten, or the files must be overwritten.

[rcarpa]

Do we want to implement the documented behavior; or to fix documentation ?

[mlassnig]

I think pilot expects this behaviour, so again, asking @PalNilsson :-)

[bziemons]

I implemented the other in #3737 and I missed the cli documentation, probably because it was marked a "bug". I am expecting the files to be overwritten, if they differ.

[rcarpa]

So we want to overwrite files if they differ; but skip them if their checksum is correct ?

[bziemons]

that's my understanding, yes. CC: @bari12

[rcarpa]

Seems logical to me. It is probably worth also doing the same thing in the case when "--no-subdir" is not set. To have a consistent behavior. If file present but checksum different : redownload. If checksum correct, don't do anything.

[bari12]

Yes, consistent behaviour would be good here I think. --no-subdir or not should not change that.