`exp show` is slow: performance issue in dulwich

[karajan1001]

Bug Report

Sometimes dvc exp show will take more than 10s to get a response, use Cprofile we notice that the time was costed in _load_file_contents in dulwich ,

related cprofile my.prof.zip

Description

Reproduce

Expected

Environment information

Output of dvc doctor:

$ dvc doctor

Additional Information (if any):

[karajan1001]

Besides the initialization progress cost of more than 30 seconds for the demo project.

while the cli git clone command only take 1 second

[~/test] → time g clone vscode-dvc vscode                                                                                                                                                          [ins][21:03:51]
Cloning into 'vscode'...
done.
Note: switching to 'f4c9beac22c5a40c497cf302c134ef83c63278cd'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

git clone vscode-dvc vscode  0.16s user 0.76s system 91% cpu 1.009 total
[~/test] →                                                                                                                                                                                         [ins][21:04:00]

[pmrowla]

Besides the initialization progress cost of more than 30 seconds for the demo project.

while the cli git clone command only take 1 second

These two operations aren't the same. We are not doing git clone when we init the experiment workspace, we are initing a new (empty) git repo and then git push into it.

The comparable operation would be

git init vscode
cd vscode-dvc
git push /absolute/path/to/vscode HEAD:HEAD

Also, CLI git clone from one local repo into another by default only generates hardlinks from the source repo's .git/objects directory into the destination repo's .git/objects directory (it does not create an independent copy of the git data).

We could consider doing the same optimization for local exp runs when possible (the tempdir .git directory has to be on the same filesystem as the original .git directory, and the fs has to support hardlinks), but using hardlinks this way is not a general (machine/filesystem independent) solution for initializing temp experiment workspaces

[karajan1001]

Takes about 1.16s.

 time git push ../vscode/ HEAD:refs/head/HEAD                                                                                                    -1-[ins][8:43:55]
Enumerating objects: 34763, done.
Counting objects: 100% (34763/34763), done.
Delta compression using up to 10 threads
Compressing objects: 100% (8562/8562), done.
Writing objects: 100% (34763/34763), 44.23 MiB | 74.38 MiB/s, done.
Total 34763 (delta 25344), reused 34733 (delta 25318), pack-reused 0
remote: Resolving deltas: 100% (25344/25344), done.
To ../vscode/
 * [new reference]       HEAD -> refs/head/HEAD
git push -f ../vscode/ HEAD:refs/head/HEAD  1.16s user 0.37s system 133% cpu 1.151 total

[pmrowla]

@karajan1001 do you still see the same performance problem after garbage collecting your repo

[karajan1001]

@karajan1001 do you still see the same performance problem after garbage collecting your repo

That one still exists.

[dberenbaum]

@karajan1001 Do you have ideas on how to fix it?

[pmrowla]

@dberenbaum we discussed it and this probably only requires filling out the libgit/pygit fetch/push implementations in scmrepo (currently we just only have dulwich implementation done, since there was never an explicit need for anything else until now).

libgit/pygit should at least be faster than dulwich, and this will be simpler than trying to dig into actual optimizations in dulwich upstream.

once that's done we can revisit this later (and consider doing upstream optimizations) if there's still a significant performance gap between using libgit/pygit and CLI git (but I don't expect there will be a significant difference)