Large Object Heap Fragmentation

[indy-singh]

Hi,

Recently we have noticed a lot of LOH allocations since migrating our platform serializer to protobuf and after spending a bit of time on it we have managed to narrow down the problem to the BufferPool.cs.

We used dotMemory and dotTrace to help us identifier the source of the issue; dotTrace points to the method ResizeAndFlushLeft:-

https://github.com/mgravell/protobuf-net/blob/a978575b9e8bc281f7341237cace35220ef0ff35/src/protobuf-net/BufferPool.cs#L49

It looks like we have certain responses that require a buffer larger than 65,536 bytes and when it goes into ResizeAndFlushLeft it gets doubled to 131,072 which is automatically allocated on the LOH as it is greater than 85,000 bytes.

We don't think the size of the buffer is a problem per-se it is the fact that they aren't being re-used. This causes repeated allocations on the LOH and causes a high frequency of fragmentation.

We managed to replicate the problem in a test harness.

We would appreciate any guidance on this issue.

Cheers,
Indy

[indy-singh]

In case it helps, you can find the test harness I put together here.

Cheers,
Indy

[mintsoft]

@singh400 We've been running protobuf in production for a while and experienced similar issues. We found there was a PR with a tweak to the bufferpool implementation which we forked and refactored to our liking which had a significant impact on LOH fragmentation.

We ran it side-by-side on different servers initially and got the following results.

This is the original buffer pool:

This is the tweaked pool that stores larger arrays:

You can see the flatness of the LOH size as the yellow line. When we have bursts of load (for example 04:00) you can see the previous behaviour causing a large amount of LOH allocation whereas the new pool is flat. The max is 1.84GB on the original implementation yet ~ 251MB on the new.

I've put our version of the bufferpool in a branch here (https://github.com/mintsoft/protobuf-net/blob/bufferpool/src/protobuf-net/BufferPool.cs), it's based heavily on (#33) however we have removed the OOM handling as we don't need it and are significantly less likely to hit any OOM condition on 64 bit machines.

I'd suggest trying that branch and see if the behaviour improves?

[indy-singh]

@mintsoft Wow, nice! Just pulled in your code very quickly into my test harness and ran a couple of quick iterations:-

Iterations	WithStockProtoBuf (MB)	WithCachedPool (MB)
10000	8.8	0.9
50000	39	4.3
100000	80	8.7
200000	160	17
300000	321	34

As you can see a massive reduction LOH allocations, quick napkin maths says around ~90% fewer LOH allocations.

Quick screenshot of dotTrace on the last test:-

WithStockProtoBuf is on the left, and WithCachedPool is on the right.

I was concerned that a change to the BufferPool.cs might result in different output from ProtoBuf. But a quick SHA1 of each output *.bin file reveals identical SHA1 hashes.

Cheers,
Indy

[lbargaoanu]

Another possible solution.

[indy-singh]

@lbargaoanu That looks pretty interesting. I did find System.Buffers.ArrayPool and just branched a version of protobuf-net that is backed by System.Buffers.ArrayPool. It shows the same ~90% reduction in LOH allocations as the version mentioned by @mintsoft.

If I get time, I might spike out another implementation using Microsoft.IO.RecyclableMemoryStream.

Cheers,
Indy

[mintsoft]

@mgravell Now I know there's a benefit that's not just limited to us, are you open to a PR of that branch to enhance the BufferPool implementation?

[indy-singh]

@mgravell Any thoughts on this and potential solutions? It seems like the branch by @mintsoft does solve this particular problem.

[indy-singh]

I have been using @mintsoft branch in production for over a week now, and I have seen the fragmentation issue severely reduced.

@mgravell I acknowledge that there at 166 open issues; but I would greatly appreciate it if we could come up with a solution to the fragmentation of the LOH and get the fix released onto NuGet so that the entire protobuf-net community can benefit.

Many thanks,
Indy

[lbargaoanu]

Most of those issues are not closed, rather than open :)

[mintsoft]

@singh400 In that spirit I've put a PR here: #307

[mintsoft]

@singh400 I've also released that fork to NuGet: https://www.nuget.org/packages/protobuf-net.Pool/ I might expand on the pooling features to allow more control around the initial sizes and number of buffers in the pool. I think it's also key to include monitoring of the pool somehow, I'll probably integrate ETW into the allocations

[indy-singh]

Any thoughts on this @mgravell or the PR by @mintsoft?

Cheers,
Indy