checkout: figure out whether to link or not

[skshetry]

Requires iterative/dvc#10513 for this to work.

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 35.71429% with 36 lines in your changes missing coverage. Please review.

Project coverage is 67.29%. Comparing base (081a1d8) to head (39ae650).
Report is 23 commits behind head on main.

Files	Patch %	Lines
src/dvc_data/hashfile/checkout.py	12.19%	36 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #548      +/-   ##
==========================================
+ Coverage   62.98%   67.29%   +4.30%     
==========================================
  Files          62       65       +3     
  Lines        4342     4785     +443     
  Branches      740      803      +63     
==========================================
+ Hits         2735     3220     +485     
+ Misses       1448     1377      -71     
- Partials      159      188      +29     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[dberenbaum]

This os call is still a bottleneck. Here's the profile for the 1.5 million files in the imagenet 2017 annotations:

add-relink-diff.prof.zip

For this to be really worth it, I think we would at least have to parallelize so these are less of a blocker or else rely on metadata to determine if they are the same.

[dberenbaum]

Opened #549 that adds a thread pool. Feel free to reimplement a different way if you want, but this has a much more reasonable profile:

add-relink-diff-thread.prof.zip

[skshetry]

Here's the profile for the 1.5 million files

I downloaded that, extracted that file. There were tar.gz files inside again, which I re-extracted. And I deleted the existing .tar.gz file.

I see following stats that says 1.07 million files. Did I do anything wrong?

Number of files: 1,077,367 (reg: 1,073,739, dir: 3,628)

---

I know macOS filesystem operations are slow, but I did not know they were this terrible.
On my machine (Linux/Intel), _check_relink finishes in 5s - 6s. First time I did dvc add, it took ~9mins, and repeated dvc add takes <1min 30s.

I can drop one more stat calls and use metadata. But I need to stat the path to object in the cache to compare inodes (we don't have cache metadata). That drops to 3s for me. :)

[skshetry]

Btw, I changed this to be strictly metadata comparison now. The inode data comes from when we check for object integrity with HashFileDB.check() (it's a bit hacky but works).

[dberenbaum]

On my machine (Linux/Intel), _check_relink finishes in 5s - 6s. First time I did dvc add, it took ~9mins, and repeated dvc add takes <1min 30s.

🤯 I just spun up a linux vm and still got 40+ minutes for the same files with this PR. I guess it shows how much the filesystem impacts performance, but I think you have done what you can with this PR.

[skshetry]

Could you please share how much time _determine_files_to_relink takes on either machines? I am just curious what it costs.

You can add @funcy.print_durations at the top of the function, and then dvc add ....

---

I guess it shows how much the filesystem impacts performance

This is what I see in my machine for posix.stat.

ncalls	tottime	percall	cumtime	percall	filename:lineno(function)
354974	0.9792	2.758e-06	0.9792	2.758e-06	~:0(<built-in method posix.stat>)

Compared that to what I see in yours:

ncalls	tottime	percall	cumtime	percall	filename:lineno(function)
11907017	2235	0.0001877	2235	0.0001877	~:0(<built-in method posix.stat>)

which is 68x slower.

Here's my cprofile for adding 1M files: add.prof.zip

[dberenbaum]

Could you please share how much time _determine_files_to_relink takes on either machines? I am just curious what it costs.

The last run I did on my mac with this PR, this only took ~4 seconds. Most of the time is spent on stat calls as you noted.