investigate perf regressions

[efiop]

E.g. dvc get seems to have jumped significantly.

Probably related to persistent index cache.

[dberenbaum]

Putting my "real-world" timings below using our public data.

2.45.1

$ time dvc import git@github.com:iterative/dataset-registry.git use-cases/cats-dogs/data/validation
dvc import git@github.com:iterative/dataset-registry.git   2.83s user 0.94s system 43% cpu 8.571 total

main

$ time dvc import git@github.com:iterative/dataset-registry.git use-cases/cats-dogs/data/validation
dvc import git@github.com:iterative/dataset-registry.git   561.23s user 170.47s system 57% cpu 21:19.81 total

[efiop]

With dvc-bench-es tiny dataset seeing old dvc transferring 14 files, while new (unreleased) one is transferring 90. Couldn't pinpoint the problem yet, likely somewhere in transfer batching changes (duh), but can't put my finger on anything yet. Looking into it...

[skshetry]

The copy is batched, so the continue makes it loop infinitely.
treeverse/dvc-objects@a0a1739

[efiop]

@skshetry Indeed, great catch! Do you want to submit a patch?

I guess not, so created treeverse/dvc-objects#202