Memory leaks

[jchiu0]

Good day!

We are from dgraph again. We have been using your implementation of a lock-free hash.

We wanted to follow up from:
#5

Since the fix, it seems that we have been seeing memory leaks. We use the hash as a caching mechanism. When there is a miss, we load extra stuff into memory. We wonder if you may have some insights into why the changes / fix might have caused the memory leaks?

Thank you so much.

Jay

[zond]

First a disclaimer: I haven't looked hard at the gotomic lib in several years.

It was really nice that you reported the race problems - when I created the lib there was no race detector IIRC, which let me miss a lot of necessary synchronization. You reporting the problems made me run the tests with the race detector on, and then simply add a lot of atomic.Load/StoreXXX calls where the race detector complained.

That this created memory leaks sounds very odd indeed. The algorithm is exactly the same, the only difference (unless I am greatly confused) is how memory is accessed.

I guess it's conceivable that the GC treats things differently depending on how they are accessed, but I've been under the impression that the GC only handles how references are stored, not how they are used.

In short, I don't understand how the latest changes should cause leaks - either the leaks should have been there before, or they should have been caused by something else...

Have you run the memory profiler? Have you changed anything else at the same time?

[jchiu0]

Hi Martin,

Thanks for the fast reply. We appreciate any help. Disclaimer is wholeheartedly accepted..

Yes, the algorithm is the same. What is puzzling is that on my desktop, it doesn't always happen. When it does, it will gobble up all 64G of my RAM. The bug is not very reproducible. On some test machine, it seems to be dying due to some other reason (https://discuss.dgraph.io/t/dissecting-the-dgraphloader-memory-usage/855/2). It also happened on some other machine (laptop of a colleague). I will need to do more runs with memory profiling....

The way we clear memory for gotomic hash is just through this one line:
https://github.com/dgraph-io/dgraph/blob/master/posting/lists.go#L308

I wonder if it is because we are adding the same key that we are trying to delete all at the same time..

Jay

[zond]

I looked at the code again, and realized that the hash removes elements lazily.

The list has only one remove function, Pop(), which does remove() that first inserts a delete marker in the list, then forces the marked element to be removed by doing next() on it's predecessor (the list head).

The hash just calls doRemove() on the element it wants gone, but never next() on the element before.

I just pushed c442ca1 that does a next() on the element found to the left of the one removed. This should force a removal of the actual element and its deleted marker.

Check if that helps!

[jiaweichiu]

Hi zond (sorry, I am not sure how to address you),

A colleague of mine (Pawan) did a memory profile:
https://dl.dropboxusercontent.com/u/99140533/mem.svg

It seems to suggest that the gotomic hash allocates a lot of memory. I believe posting.NewList is the one that allocates memory for values of the hash, so the memory allocated by gotomic hash is something else, say the keys and internal structures to represent buckets, linked lists etc I suppose.

*Wonder if some optimization can be done here to reduce the memory usage? I should perhaps file this as a separate issue.

With respect to the original issue, I haven't seen it occur again yet. However, I will still want to try out your change and do some memory profiling again.

As always, thank you for your speedy replies!

JC
(on behalf of Dgraph team)

[zond]

Hey JC, sorry for not introducing myself earlier!

My name is Martin Bruse. Nice to meet you guys :D

In your profile, it seems that most of the in-use memory is connected to the String() functions of the gotomic data types - are you perchance printing the hash regularly, or doing some other introspection that calls String() on them?

But yeah, the atomic hash isn't as efficient - space- or time-wise - as a regular map[...]....

If you haven't seen the original issue lately, I'll just close this ticket. Re-open if it occurs again!

My usual speedy replies will be slightly slower now, since my second child was born 2 days ago :)

best,
Martin

[jchiu0]

Thank you Martin for all the help thus far! And all the best with your second child!