OOM error on small repo when cloning/pull

[DUOLabs333]

I have a very small repo (~1Mb), yet when cloning, git-remote-https gets killed because the system runs out of memory. It's only this repo, all others work fine.

[jonashaag]

Can you share the repo?

[jelmer]

Which version of Dulwich are you using? There have been some fixes recently that improve the strategy for delta resolution - significantly improving memory usage on certain kinds of repository.

[DUOLabs333]

@jonashaag It's a private repository.

[masatake]

Do you use ctags (in klaus)?

[DUOLabs333]

No, I don't think so. How would I know if I do?

[DUOLabs333]

Which version of Dulwich are you using? There have been some fixes recently that improve the strategy for delta resolution - significantly improving memory usage on certain kinds of repository.

I'm using version 0.20.35, which seems to be the latest version.

[jonashaag]

Maybe you can send the private repo via email so I can check it out

[DUOLabs333]

homebrew-qemu-virgl-git.tar.gz

[jonashaag]

Thanks, will have a look tomorrow.

[DUOLabs333]

Correction: this is happening on all but a few of my repos.

[jonashaag]

@jelmer can you look into this? I'm not sure what's going on here. It seems to hang inside git-remote-http but I can't find any meaningful differences in the info/refs?service=git-upload-pack compared to what GitHub responds. (Except for an extra 0x20 character in Dulwich's reponse which doesn't seem to make any difference)

[DUOLabs333]

@jonashaag So you can recreate this issue with the repo I sent? Does this happen with any of your repos?

[jonashaag]

Yes it happens with my repos as well. I wonder if it has something to do with a recent Git release but an older version of Git had the same issue.

What’s the OS you run git on? I tried macOS only.

[DUOLabs333]

This is with Arch Linux.

[DUOLabs333]

Where is the code for that stored? Maybe I can take a look at it? (Also, maybe you should think of adding an option to use git-http-backend instead).

[jonashaag]

The memory leak is in the git-remote-http code

[DUOLabs333]

Wouldn't that indicate somethings wrong on the server's side?

[jonashaag]

Yes, and/or in Git.

[jelmer]

FWIW a number of memory improvements have gone into Dulwich' server gode recently.

[DUOLabs333]

Ok, I'll try it out.

[DUOLabs333]

The problem seems to remain (dulwich 0.20.45).

[jonashaag]

@jelmer any idea what's going on here? I'm facing the same problem now on macOS with the latest Dulwich and Git.

[jonashaag]

It starts to happen with Git 2.28. 2.27 is fine.

[jonashaag]

Is it related to this? https://github.com/git/git/blob/47ae905ffb98cc4d4fd90083da6bc8dab55d9ecc/Documentation/RelNotes/2.28.0.txt#L91-L93

[jonashaag]

@jelmer note that the memory leak is in the client

[jonashaag]

Profile of the infinite loop in the Git client

[jelmer]

Can you reproduce this with git on the server side?

[jonashaag]

It works fine with git daemon --verbose --port=8080 --base-path=. --export-all.

Anything else I should check?

[jonashaag]

I reported a Git bug.

[jonashaag]

@jelmer FYI see discussion here. Bug in Git that is about to be fixed. https://lore.kernel.org/git/39035D34-8548-44B0-BBBB-5C36B3876C4A@lophus.org/

[jelmer]

Interesting - thanks for the link & background.