Hardlinks not correctly detected on SMB network share (NTFS?)

[rxxg]

Using DVC version 0.80, with repo and cache both on the same network drive.
Windows setup.

I'm testing the unprotect command, but I assume other commands relying on the same check will also exhibit strange failures.

PS Q:\dvc-test> dvc --version
0.80.0
PS Q:\dvc-test> cat .dvc\config
[cache]
dir = 'Q:\DVC cache (test)'
type = "reflink,symlink,hardlink,copy"
protected = true
PS Q:\dvc-test> dir


    Directory: Q:\dvc-test


Mode                LastWriteTime         Length Name
----                -------------         ------ ----
d-----       06/01/2020     16:58                .dvc
-a----       06/01/2020     14:57             24 .gitignore
-a----       06/01/2020     14:57            155 big_file.dvc
-ar---       27/12/2019     16:42      134217728 big_file

PS Q:\dvc-test> fsutil.exe hardlink list big_file
Error:  The request is not supported.

As you can see, I checked out my files from git at 14:57 today; dvc checkout created a protected link to the original 128Mb file created 10 days ago, rather than a copy (as requested), but Windows can't tell me that it is a link. Trust me, it is 😉

PS Q:\dvc-test> dvc unprotect big_file
PS Q:\dvc-test> dir


    Directory: Q:\dvc-test


Mode                LastWriteTime         Length Name
----                -------------         ------ ----
d-----       06/01/2020     16:58                .dvc
-a----       06/01/2020     14:57             24 .gitignore
-a----       06/01/2020     14:57            155 big_file.dvc
-a----       27/12/2019     16:42      134217728 big_file

Strange - the unprotect operation was very fast, and the creation date of the file is still last year rather than today.

PS Q:\dvc-test> echo Hello >> big_file
PS Q:\dvc-test> dvc status
WARNING: corrupted cache file '..\DVC cache (test)\1d\cca63ad430e16fa12716d1a9bb3a6c'.
big_file_3.dvc:
     changed outs:
                                                                                                                       not in cache:       big_file_3

Sure enough, modifying big_file corrupts the cache, since I was not modifying a copy.

[pared]

@rxxg I am unabe to reproduce the issue,
can I ask you to run following scirpt?

rmdir /s repo
mkdir repo
pushd repo
git init --quiet
dvc init -q
dvc config cache.type hardlink
dvc config cache.protected true
git commit -am "init dvc"
fsutil file createnew data 10485760
dvc add data
git add .gitignore data.dvc
git commit -am "add data"
fsutil hardlink list data
dvc unprotect data
fsutil hardlink list data
echo hello >> data
dvc status
popd

Does the status display corrupted cache WARNING?

[EDIT]
Also, can I ask you to provide output of dvc version command? (note that its without --)

[EDIT2]
Sorry, forgot its NFS drive, let me try to reproduce that again.

[pared]

@rxxg Also, as a temporary workaround you can change cache type to copy (dvc config cache.type copy) and use dvc checkout --relink big_file.dvc.

[rxxg]

@pared Thanks for looking at this. Yes, the use of a network drive (repo and cache) is essential.

(I had been using copy cache but it is very slow for our use case since the copy that DVC does involves reading then writing to the network drive in 16k chunks. Native Windows copy is of the order of 3 seconds for a 128Mb file, 30 seconds for shutil.copyfileobj. There may be a separate bug report or PR for this.)

[rxxg]

For the record:

Q:\repo> git init --quiet
Q:\repo> dvc init -q
Q:\repo> dvc config cache.type hardlink
WARNING: You have changed the 'cache.type' option. This doesn't update any existing workspace file links, but it can be done with:
             dvc checkout --relink
Q:\repo> dvc config cache.protected true
Q:\repo> git commit -am "init dvc"
[master (root-commit) 7e8ba77] init dvc
 2 files changed, 12 insertions(+)
 create mode 100644 .dvc/.gitignore
 create mode 100644 .dvc/config
Q:\repo> fsutil file createnew data 10485760
File Q:\repo\data is created
Q:\repo> dvc add data
100% Add|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████|1.00/1.00 [00:01<00:00,  1.48s/file]

To track the changes with git, run:

     git add .gitignore data.dvc
Q:\repo> git add .gitignore data.dvc
Q:\repo> git commit -am "add data"
[master d7e862f] add data
 2 files changed, 8 insertions(+)
 create mode 100644 .gitignore
 create mode 100644 data.dvc
Q:\repo> fsutil hardlink list data
Error:  The request is not supported.
Q:\repo> dvc unprotect data
Q:\repo> fsutil hardlink list data
Error:  The request is not supported.
Q:\repo> echo hello >> data
Q:\repo> dvc status
WARNING: corrupted cache file '.dvc\cache\f1\c9645dbc14efddc7d8a322685f26eb'.
data.dvc:
     changed outs:
                                                                                                                                                                                           not in cache:       data
Q:\repo> dvc version
DVC version: 0.80.0
Python version: 3.7.1
Platform: Windows-10-10.0.16299-SP0
Binary: False
Package: pip
Cache: reflink - False, hardlink - True, symlink - False

[pared]

@rxxg Ok, thank you very much, I am trying to reproduce it on my machine.

[efiop]

@rxxg Could you please install psutil with pip install psutil and then run dvc version again and show us the output?

Side note for us: need to improve the way dvc version tests for link types by doing additional check for the created links. E.g. create hardlink and then do a sanity check with System.is_hardlink.

[rxxg]

Sure.

DVC version: 0.80.0
Python version: 3.7.1
Platform: Windows-10-10.0.16299-SP0
Binary: False
Package: pip
Cache: reflink - False, hardlink - True, symlink - False
Filesystem type (cache directory): ('NTFS', 'Q:\\')
Filesystem type (workspace): ('NTFS', 'Q:\\')

[efiop]

@rxxg I see that it is reporting NTFS, but you were saying you are on NFS. Was it a typo or am I missing something?

[efiop]

@rxxg Btw, is it your work machine or your personal one? We've seen something similar on NTFS in #2831 , but weren't able to find the cause for such a strange FS behavior at that time.

[rxxg]

Sorry, typo 😳 Windows NTFS network share
It's my work machine, so I have zero control over the servers and even finding out info about the hardware/network protocol is hard work.
I had locking failures which came from the same cause (#2944) but things are fine since the change to the locking system.