Huge amount of memory used.

[ganju001]

Prerequisites

• I have written a descriptive issue title
• I have searched issues to ensure it has not already been reported

GitVersion package

GitVersion.Portable

GitVersion version

6.0.1

Operating system

Windows

What are you seeing?

Hello community,

We have mono-repo that is more that few gigabytes in size. We were using gitversion 5.x and it was calculating
the version in 15 minutes, after updating to gitversion 6.x, I got huge memory (RAM) usage: 40GB+ (yes 40, no mistake) and more than 2 hours...

Any suggestions?

What is expected?

I expect it will calculate version with reasonable RAM usage and time.

Steps to Reproduce

Internal repo..

RepositoryFixture Test

No response

Output log or link to your CI build (if appropriate).

[HHobeck]

I would suggest to wait until version v6.2 has been released. After that please analyze your execution and create a memory dump. You need to narrow down the locaction where the memory is allocated or the CPU time is used. I'm going to move this to the discussion forum.

Cheers

[ganju001]

@HHobeck I have a memory dump, I can try to give some insights, the problem is that LibGit2Sharp creates objects which takes gigabytes, gitversion wraps those objects (commits, collections) and those take gigabytes too, I Saw 400M+ of references and 300M+ of objects created.

Normal branches (non DevOps PR ones) takes ~1minute and 7GB RAM to calculate (that is kind of acceptable); but the PR ones are taking ages and more that expected RAM..

[HHobeck]

@HHobeck I have a memory dump, I can try to give some insights, the problem is that LibGit2Sharp creates objects which takes gigabytes, gitversion wraps those objects (commits, collections) and those take gigabytes too, I Saw 400M+ of references and 300M+ of objects created.

Normal branches (non DevOps PR ones) takes ~1minute and 7GB RAM to calculate (that is kind of acceptable); but the PR ones are taking ages and more that expected RAM..

We had some performance improvement in the upcoming version. Please wait with the analyzis. You say that LibGit2Sharp uses to much memory... Maybe we have a memory leak there. Normally the calls to LibGit2Sharp are wrapped and the results are cached. A code review might be an idea having this in mind.

[ganju001]

@HHobeck Few questions:

• Do we have some branch that I can try?
• I doubt it is memory leak, as the memory usage goes up and down many times. It seems like a caching. And having 300 000 000+ objects in memory for cache is an issue for sure. Too many strings are created too...

This is somewhere in the middle of calculation:

In example:
Commit person name/email strings takes 1.3GB. Some names are repeated 20k+ times - duplicates.

[HHobeck]

Do we have some branch that I can try?

The main branch includes the latest development version. If you want to do changes please fork the repository and create a feature branch. By creating a pull requst you can merge it back to the main branch of gitversion.

Commit person name/email strings takes 1.3GB. Some names are repeated 20k+ times - duplicates.

This properties are not really necessary for calculating the next version number. I'm not sure if we have it in control

[ganju001]

Alright, I will make my changes again, and we will see how it goes

[HHobeck]

Alright, I will make my changes again, and we will see how it goes

If you not mind please create a fork of gitversion and do this changes in your github space. This would give us the possibility to review the changes. Also it can be used as a basis for any discussions related to this topic. Do you know how to create a fork?

[ganju001]

@HHobeck the change

ganju001@19df40e

after it I get same memory hungry issue as I talked before

[HHobeck]

Okay thank you very much.

The QueryBy method is accessing the innerCollection. This might be the reason why the objects are still allocated and not reused:

GitVersion/src/GitVersion.LibGit2Sharp/Git/CommitCollection.cs

Lines 25 to 36 in dd9434b

	public IEnumerable<ICommit> QueryBy(CommitFilter commitFilter)
	{
	var includeReachableFrom = GetReacheableFrom(commitFilter.IncludeReachableFrom);
	var excludeReachableFrom = GetReacheableFrom(commitFilter.ExcludeReachableFrom);
	var filter = new LibGit2Sharp.CommitFilter
	{
	IncludeReachableFrom = includeReachableFrom,
	ExcludeReachableFrom = excludeReachableFrom,
	FirstParentOnly = commitFilter.FirstParentOnly,
	SortBy = (LibGit2Sharp.CommitSortStrategies)commitFilter.SortBy
	};
	var commitLog = ((IQueryableCommitLog)this.innerCollection).QueryBy(filter);

@arturcic and @asbjornu any ideas?

[arturcic]

LibGit2Sharp.CommitFilter is a LibGit2Sharp class that are is used at the libgit2 level. Not entirely sure how it's used in the libgit2 layer (which is a native code). Ideally we need to do the filtering ourselves in-memory on the already fetched commit collection.

QueryBy itself returns a Commit collection (IQueryableCommitLog). So not sure how we need to handle this