Memory leak when useing embedded documents

[TheHett]

Bug Report

| Version | 2.0.3

Summary

I ran into a memory leak and it occurs when using EmbedMany annotation.

/**
 * @ODM\Document(collection="file_keyfames_hash")
 */
class FileKeyframesHash
{
    /**
     * @var Collection<FileKeyframesHash>
     * @ODM\EmbedMany(name="hashes_4", targetDocument=KeyframesHash::class)
     */
    private $hashes4;
}

/**
 * @ODM\EmbeddedDocument()
 */
class KeyframesHash
{
    /**
     * @var string
     * @ODM\Field(name="hash", type="bin")
     */
    private $hash;

    // ... getters/setters
}

        $cursor = $dm->createQueryBuilder(FileKeyframesHash::class)
            ->getQuery()
            ->execute();

        $count = 0;
        foreach ($cursor as $e) {
            /** @var FileKeyframesHash $e */
            $count++;
            if ($count % 10 === 0) {
                echo "Processed: {$count}", PHP_EOL;
                echo "Managed: {$dm->getUnitOfWork()->size()}", PHP_EOL;
                echo Helper::bytesHumanize(memory_get_usage()), PHP_EOL;
                echo PHP_EOL;
            }
            $dm->detach($e);
            unset($e);
        }

The problem occurs event when used $dm->clear() instead of $dm->detach($e)

UPD: HERE I PROVIDED TEST #2162 (comment)

example of my script output:

Processed: 100
Managed: 101
472.7 MB

Processed: 110
Managed: 111
538.9 MB

Processed: 120
Managed: 121
592.3 MB

Processed: 130
Managed: 131
684.9 MB

Processed: 140
Managed: 141
766.0 MB

Processed: 150
Managed: 151
851.1 MB

Processed: 160
Managed: 161
903.2 MB

Processed: 170
Managed: 171
967.1 MB

PHP Fatal error:  Allowed memory size of 1073741824 bytes exhausted (tried to allocate 20480 bytes) in /usr/local/app/vendor/doctrine/mongodb-odm/lib/Doctrine/ODM/MongoDB/Iterator/HydratingIterator.php on line 106
PHP Stack trace:
PHP   1. {main}() /usr/local/app/yii:0
PHP   2. yii\console\Application->run() /usr/local/app/yii:32
PHP   3. yii\console\Application->handleRequest() /usr/local/app/vendor/yiisoft/yii2/base/Application.php:386
PHP   4. yii\console\Application->runAction() /usr/local/app/vendor/yiisoft/yii2/console/Application.php:147
PHP   5. yii\console\Application->runAction() /usr/local/app/vendor/yiisoft/yii2/console/Application.php:180
PHP   6. app\commands\TestController->runAction() /usr/local/app/vendor/yiisoft/yii2/base/Module.php:528
PHP   7. app\commands\TestController->runAction() /usr/local/app/vendor/yiisoft/yii2/console/Controller.php:164
PHP   8. yii\base\InlineAction->runWithParams() /usr/local/app/vendor/yiisoft/yii2/base/Controller.php:157
PHP   9. call_user_func_array:{/usr/local/app/vendor/yiisoft/yii2/base/InlineAction.php:57}() /usr/local/app/vendor/yiisoft/yii2/base/InlineAction.php:57
PHP  10. app\commands\TestController->actionExportHashes() /usr/local/app/vendor/yiisoft/yii2/base/InlineAction.php:57
PHP  11. Doctrine\ODM\MongoDB\Iterator\CachingIterator->next() /usr/local/app/commands/TestController.php:121
PHP  12. Generator->next() /usr/local/app/vendor/doctrine/mongodb-odm/lib/Doctrine/ODM/MongoDB/Iterator/CachingIterator.php:89
PHP  13. Doctrine\ODM\MongoDB\Iterator\CachingIterator->wrapTraversable() /usr/local/app/vendor/doctrine/mongodb-odm/lib/Doctrine/ODM/MongoDB/Iterator/CachingIterator.php:89
PHP  14. Doctrine\ODM\MongoDB\Iterator\HydratingIterator->next() /usr/local/app/vendor/doctrine/mongodb-odm/lib/Doctrine/ODM/MongoDB/Iterator/CachingIterator.php:156
PHP  15. Generator->next() /usr/local/app/vendor/doctrine/mongodb-odm/lib/Doctrine/ODM/MongoDB/Iterator/HydratingIterator.php:71
PHP  16. Doctrine\ODM\MongoDB\Iterator\HydratingIterator->wrapTraversable() /usr/local/app/vendor/doctrine/mongodb-odm/lib/Doctrine/ODM/MongoDB/Iterator/HydratingIterator.php:71

[malarzm]

Probably detach is not unsetting embedded documents' own identity map. Theoretically it'll be easy to fix unless we did that specifically because of spl_object_id collisions.

[Steveb-p]

@malarzm Isn't it related to QueryBuilder caching results?

[malarzm]

could be, I'm replying from the top of my head :)

[alcaeus]

@malarzm Isn't it related to QueryBuilder caching results?

QueryBuilder does not cache results, as it doesn't even have any. Results are only stored in UnitOfWork through identity map. However, detach operations cascade for embedded relationships, so that every embedded document is properly detached. I was also able to reproduce this in a test.

@TheHett are there any other relationships (maybe references) that aren't detached? The continuously increasing size of the unit of work suggests that.

[Steveb-p]

QueryBuilder does not cache results, as it doesn't even have any. Results are only stored in UnitOfWork through identity map.

I can see CachingIterator in the provided stacktrace, which has caching in it.

[TheHett]

are there any other relationships (maybe references) that aren't detached? The continuously increasing size of the unit of work suggests that.

Yes, but i tried to remove different fields, and made sure that this is field $hashes4.
In addition, this can be seen in the rapid increase of memory consumption (the field $hashes4 contains very large array of bin data and the script falls after ~150 iterations despite 1gb memory limit )

[Steveb-p]

@TheHett try applying code changes from this PR #2116

[alcaeus]

I can see CachingIterator in the provided stacktrace, which has caching in it.

That is due to the Query class itself, not the builder. The builder only assembles the query array and passes it on to the query object when you run getQuery(). Semantic difference, but important nonetheless.

[Steveb-p]

Sorry 😿, I meant Query class.

Anyway, I'm practically sure that the CachingIterator is at fault here. Applying fix from the PR and setting Query as ->rewindable(true) should help.

[alcaeus]

Yep, CachingIterator may very well be the culprit. 😢
Detaching entities works as expected, but there's an additional reference stored in the iterator instance, so it's never garbage collected. Still, doesn't explain why the UoW size keeps growing 🤔

[Steveb-p]

doesn't explain why the UoW size keeps growing

I was wondering as well. Maybe embedded documents are still kept, even if parent was detached 🤔 ? I was sure there were some tests that covered this though (don't have time to look for it atm).

[alcaeus]

I was wondering as well. Maybe embedded documents are still kept, even if parent was detached 🤔 ? I was sure there were some tests that covered this though (don't have time to look for it atm).

I explicitly tested that:

public function testDetachCascadesToEmbeddedDocuments()
{
    $user = new User();
    $user->addPhonenumber(new Phonenumber('123'));
    $user->addPhonenumber(new Phonenumber('234'));

    $this->dm->persist($user);
    $this->dm->flush();

    $this->dm->clear();

    $user = $this->dm->find(User::class, $user->getId());
    $this->assertNotNull($user);
    $this->assertCount(2, $user->getPhoneNumbers());

    $this->assertSame(3, $this->dm->getUnitOfWork()->size());

    $this->dm->detach($user);

    $this->assertSame(0, $this->dm->getUnitOfWork()->size());
}

Test passes successfully.

[TheHett]

try applying code changes from this PR #2116

@Steveb-p ok, I can only do it tomorrow at the office