perf: copy cache operations very slow on network drive

[rxxg]

Execute the following script on a Windows network drive

mkdir repo
cd repo
git init --quiet
dvc init -q
dvc config cache.type copy
measure-command { fsutil file createnew data 134217728 }
# Probably less than 1 second
measure-command { dvc add data }
# Around 30 seconds on our setup

Digging into the problem a little bit it seems that the cache copy operations end up in shutil.copyfileobj which reads the file to be copied into memory in 16kb chunks before writing out again. Unless the network is very local this is always going to be a performance killer.

The situation might be better with Python 3.8 (#3033), but it would be good to ease the pain until DVC supports this version, and even then some users will not be able to upgrade straightaway.

[efiop]

Hi @rxxg !

How did you reach the conclusion about blocksize in shutil.copyfileobj? Also, could you please elaborate on what will happen in 3.8 with it? Sorry, not aware of those upcoming changes 🙂

Also, it is important to note that dvc is not only copying the file, but also is computing an md5 for it to be able to facilitate it in its cache. Could you try running md5sum(not sure what is the alternative command for windows) and time it to see how much time it takes compared to dvc add? And also timing cp data data_copy would be interesting to see as well, just so we could understand better the situation.

[rxxg]

Hi @efiop,
I just looked at the 3.7 source code for shutil.copyfile and copyfileobj - by default the chunks are 16*1024 bytes.
There were a number of patches in this area for 3.8, for example issue 33671, issue 36103 and issue 36610 which mean that the current code is much more complete.

A few small projects exist to add or complement this functionality - e.g. pyfastcopy and speedcopy. I have started playing with speedcopy since I am interested in Windows network drive performance but it is early code and doesn't seem to work with relative paths so it will need patching to work for DVC.

The ideal for me would be to have full server-side copy - although I am not sure whether my file servers actually support this!

[rxxg]

Here are the detailed perf figures for my SSD and the network drive

(dvc_master) PS C:\Users\rxg\repo> git init --quiet
(dvc_master) PS C:\Users\rxg\repo> dvc init -q
(dvc_master) PS C:\Users\rxg\repo> dvc config cache.type copy
WARNING: You have changed the 'cache.type' option. This doesn't update any existing workspace file links, but it can be done with:
             dvc checkout --relink
(dvc_master) PS C:\Users\rxg\repo> measure-command { fsutil file createnew data 134217728 }
TotalMilliseconds : 58.9091

(dvc_master) PS C:\Users\rxg\repo> measure-command { Get-FileHash -Algorithm MD5 data }
TotalMilliseconds : 575.5359

(dvc_master) PS C:\Users\rxg\repo> Measure-Command { copy data data.copy }
TotalMilliseconds : 478.34

(dvc_master) PS C:\Users\rxg\repo> Measure-Command { dvc add data }
100% Add|█████████████████████████████████████████████████████████████████████████|1.00/1.00 [00:02<00:00,  2.03s/file]

TotalMilliseconds : 3812.4002

So much for the local disk, let's try the network drive.

(dvc_master) PS C:\Users\rxg\repo> Q:
(dvc_master) PS Q:\> mkdir repo
(dvc_master) PS Q:\> cd repo
(dvc_master) PS Q:\repo> git init --quiet
(dvc_master) PS Q:\repo> dvc init -q
(dvc_master) PS Q:\repo> dvc config cache.type copy
WARNING: You have changed the 'cache.type' option. This doesn't update any existing workspace file links, but it can be done with:
             dvc checkout --relink
(dvc_master) PS Q:\repo> measure-command { fsutil file createnew data 134217728 }
TotalMilliseconds : 81.9581

(dvc_master) PS Q:\repo> measure-command { Get-FileHash -Algorithm MD5 data }
TotalMilliseconds : 2586.4955

(dvc_master) PS Q:\repo> Measure-Command { copy data data.copy }
TotalMilliseconds : 3032.9972

(dvc_master) PS Q:\repo> Measure-Command { dvc add data }
100% Add|█████████████████████████████████████████████████████████████████████████|1.00/1.00 [00:25<00:00,  25.1s/file]

TotalMilliseconds : 27536.3063

In both cases, DVC add is of the order of 3-5 times more expensive than native operations. (And it's interesting to note how quickly the file is initially generated in both cases.)

[rxxg]

Closed in #3135