Fix reindex memory leaks

[noirbizarre]

This PR fixes 2 memory leak on search indexing.

The memory profile have been sampled with the following configuration:

• CPU: Inter Core I7 7500U
• RAM: 16G
• OS: Archlinux
• Datasets: 10000

Tooling

The profiling script is: https://gist.github.com/noirbizarre/7ea0b0526814131752e6902a4254df78

It requires:

• memory-profiler
• objgraph
• matplotlib
• graphviz

Run it with:

mprof run -o output.dat profiling.py

Render graph with:

mprof plot output.dat -t "Title" -o output.png

Memory profile from current master

Max memory usage: ~4G

udata SlugField

The udata SlugField instances were keeping a reference to their parent document using self.instance. This reference was stored in a class instance which is instanciated once on document definition and so the garbage collector was never able to collect document instances using a slug field (as there was always a reference on it)

This has been fixed by refactoring the SlugField population into a signal handler and not keeping a static feference on the instance.

Memory profile with the SlugField fix

Max memory usage: ~3.6G

MongoEngine querysets and no_cache()

MongoEngine default behavior is to cache any queryset just to avoid round trip on "count then query".
This behavior cause a massive memory leak while iterating over a queryset generator.

The fix is to add a no_cache() call before the loop.

This fix might be beneficial on other code parts.

Memory profile with no_cache()

Max memory usage: ~300M

MongoEngine 0.17.0

It doesn't fix the memory leak but the update was applied just to verify that it has no impact (See this PR).
As you can see, there is some meory drop and the hole reindexing take much more longer.
The update has been removed from the PR until the cause of these drops is found.

Memory profile

Max memory usage: ~300M