Memory leak when I clear the document manager #415

Closed
pierrre opened this Issue Sep 18, 2012 · 8 comments

Projects

None yet

4 participants

Contributor
pierrre commented Sep 18, 2012

Hi!

I use a fixture script that insert 3000 documents.
They are a bit complex (I use embedded documents and reference many).
I flush the DM with a "batch" system (100 documents).
After the flush, i call $dm->clear().

Here is the result of var_dump(memory_get_usage(true)/1024/1024); after clear/flush.

double(135.5)
double(135.5)
double(146.5)
double(158.25)
double(170.5)
double(182.75)
double(195.25)
double(207.25)
double(219.5)
double(231.5)
int(244)
double(256.25)
double(268.5)
double(280.75)
double(292.5)
double(305.25)
double(317.25)
double(329.5)
double(341.5)
double(353.75)
double(365.75)
double(377.75)
int(390)
double(402.25)
double(414.25)
double(426.25)
double(438.5)
double(450.75)
int(463)
double(475.5)

It's a big memory leak! :P
I checked my script, but there is nothing special.
I don't think there is a memory leak in my code.

I use $dm->getReference() in order to speed up my fixture.
I know the DM reuse document references when it's the same identifier/class (identity map)
If i don't call $dm->clear() after flush(), I have this:

double(143.75)
double(143.75)
int(136)
int(136)
int(136)
int(136)
double(139.25)
double(141.75)
int(145)
double(148.25)
int(151)
double(153.75)
double(156.25)
double(159.5)
double(162.25)
double(165.5)
int(169)
double(171.75)
double(174.75)
double(177.5)
double(180.25)
double(183.5)
double(186.5)
double(189.25)
int(192)
int(195)
double(197.75)
double(200.25)
int(203)
double(205.75)

There is still a memory leak of course, but my 3000 created documents are not removed :P
The memory leak is smaller, because the DM reuse other documents.

I think the problem is: when the document manager is cleared, all the documents are kept in memory because they have circular references.

I added a little hack in UnitOfWork->clear().
It "cleans" all documents when they are removed from the DM.

            foreach($this->identityMap as $class => $documents) {
                $reflectionClass = new \ReflectionClass($class);
                $reflectionProperties = $reflectionClass->getProperties();
                foreach($documents as $identifier => $document) {
                    foreach($reflectionProperties as $reflectionProperty) {
                        if(!$reflectionProperty->isStatic()) {
                                $reflectionProperty->setAccessible(true);
                                $reflectionProperty->setValue($document, null);
                        }
                    }
                }
            }

Here is the memory usage:

double(133.75)
double(133.5)
double(133.25)
double(130.5)
double(130.5)
double(130.5)
double(130.5)
double(130.5)
double(130.5)
double(130.5)
double(130.5)
double(130.5)
double(130.5)
double(130.5)
double(130.5)
double(130.5)
double(130.5)
double(130.5)
double(130.5)
double(130.5)
double(130.5)
double(130.5)
double(130.5)
double(130.5)
double(130.5)
double(130.5)
double(130.5)
double(130.5)
double(130.5)

No memory leak!

What do you think?

Contributor
pierrre commented Sep 18, 2012

PersistentCollection has a reference to its owner

Owner
jmikola commented Sep 18, 2012

I'm aware that memory leaks are common occurrences when using ODM, but this solution doesn't seem suitable for general use. Clearing the document manager should really just instruct UoW to stop tracking those documents. It detaches everything, but should not modify the documents themselves. In your case, you're unsetting all properties within all tracked documents before detaching them.

Contributor
pierrre commented Sep 18, 2012

I know this hack is really ugly :P

What should I do to avoid memory leak in my application?
I can't do anything, because I can't destroy documents when the DM is cleared...

Owner
jmikola commented Sep 18, 2012

This is fine for your application (you can certainly keep it in your own fork), I'm just not willing to merge it into ODM as-is.

Alternatively, a less invasive fix for your case may be to access all managed documents in UoW (perhaps that requires reflection to reach into UoW) and run your snippet manually. That would at least not require custom code within UoW's clear() method.

Contributor
pierrre commented Sep 19, 2012

I've found a way to "fix" the memory leak in my case.
Here's the "schema" of the documents I insert in Doctrine:

document => embed many => embedded document => reference many => document

When I create my document, I create a new ArrayCollection for the "embed many" field.
It is transformed to a PersistentCollection after persist/flush.

If I unset the PersistentCollection after $dm->clear(), there is no memory leak.
So, there is maybe a problem in PersistentCollection :P

If I unset both 'snapshot' and 'coll' properties in PersistentCollection (using reflection), there is no memory leak.

What do you think?

Owner
jmikola commented Sep 19, 2012

That's a great lead, thanks. I will start there when I find time to resume investigating this. I'm currently working on a GridFS bug in doctrine/mongodb and then moving on to a bug with dot separators in ODM (for things like positional operators).

@jmikola jmikola was assigned Sep 19, 2012

@jmikola $entity->set_CollectionName_(null); after $dm->clear and only with --no-debug worked for me (symfony 2.1), so I guess the problem is in DoctrineMongoDBBundle?

@jwage jwage closed this Jan 5, 2014
Owner
jwage commented Jan 5, 2014

Added a memory performance test here 6a206cd

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment