Pimcore Memory Leak on Importing thousands of Objects #3104

dpfaffenbauer · 2018-07-19T08:48:20Z

Based on complexity and data-amount of an Object Class, memory increases heavily on Importing/updating thousands of items. I have an import with around 20k Products, with a lot of informations like classifications, images, relations, etc. Around every 10th product, memory increases about 2MB. So, 20k products means a memory consumption of around:

100mb base usage (large import file needs to be kept in memory + symfony container)
+ every 10th item consumes 2 mb
= more than 4gb of ram usage.

To make this easier test-able, I created a new Pimcore installation (latest stable release, 5.2.3). Created a very simple class:

And ran imports with different amounts (used PHP7.1):

Amount	Time	Start Memory Usage	End Memory Usage	Blackfire Report
1000	~2min	16mib	94 mib	https://blackfire.io/profiles/2b2cbe9a-9eba-4de3-ac60-b1515f5db389/graph
10000	~21min	16mib	734 mib	https://blackfire.io/profiles/e814e0ac-3a0d-4c4e-9e8c-a952883f12e2/graph

So I guess you can imagine the memory consumption for 20k or more items.

Command used for importing data:

<?php

namespace AppBundle\Command;

use Pimcore\Model\DataObject\Service;
use Pimcore\Model\DataObject\Test;
use Symfony\Bundle\FrameworkBundle\Command\ContainerAwareCommand;
use Symfony\Component\Console\Helper\ProgressBar;
use Symfony\Component\Console\Input\InputArgument;
use Symfony\Component\Console\Input\InputInterface;
use Symfony\Component\Console\Output\OutputInterface;

class AppTestCommand extends ContainerAwareCommand
{
    protected function configure()
    {
        $this
            ->setName('app:test')
            ->addArgument('itemsToCreate', InputArgument::REQUIRED);
    }

    protected function execute(InputInterface $input, OutputInterface $output)
    {
        $itemsToCreate = $input->getArgument('itemsToCreate');

        $progress = new ProgressBar($output, $itemsToCreate);
        $progress->setFormat(
            '%current%/%max% [%bar%] %percent:3s%% (%elapsed:6s%/%estimated:-6s%) %memory:6s%: %message%'
        );

        for ($i = 0; $i < $itemsToCreate; $i++) {
            $progress->setMessage('Create Item with Number '.$i);

            $item = new Test();
            $item->setBlub(uniqid());
            $item->setBlub2(uniqid());
            $item->setBlubLocalized(uniqid());
            $item->setKey($i.'-'.uniqid());
            $item->setParent(Service::createFolderByPath('/blub-'.$itemsToCreate.'/'.$i % 100));
            $item->save();

            $progress->advance();
        }
    }

}

Wrapping import in a nested transactions brings following results. A bit better but not much.

Amount	Time	Start Memory Usage	End Memory Usage	Blackfire Report
1000	~1min	16mib	84mib	https://blackfire.io/profiles/2600b39a-b6e3-41b9-9ba3-0ce2142d123d/graph
10000	~20min	16mib	698mib	https://blackfire.io/profiles/78176e0b-7afe-477d-982c-387a9cc9b3d0/graph

Command used for nested transactions

<?php

namespace AppBundle\Command;

use Pimcore\Db;
use Pimcore\Model\DataObject\Service;
use Pimcore\Model\DataObject\Test;
use Symfony\Bundle\FrameworkBundle\Command\ContainerAwareCommand;
use Symfony\Component\Console\Helper\ProgressBar;
use Symfony\Component\Console\Input\InputArgument;
use Symfony\Component\Console\Input\InputInterface;
use Symfony\Component\Console\Output\OutputInterface;

class AppTestCommand extends ContainerAwareCommand
{
    protected function configure()
    {
        $this
            ->setName('app:test')
            ->addArgument('itemsToCreate', InputArgument::REQUIRED);
    }

    protected function execute(InputInterface $input, OutputInterface $output)
    {
        $itemsToCreate = $input->getArgument('itemsToCreate');

        $progress = new ProgressBar($output, $itemsToCreate);
        $progress->setFormat(
            '%current%/%max% [%bar%] %percent:3s%% (%elapsed:6s%/%estimated:-6s%) %memory:6s%: %message%'
        );

        Db::get()->beginTransaction();

        for ($i = 0; $i < $itemsToCreate; $i++) {
            $progress->setMessage('Create Item with Number '.$i);

            $item = new Test();
            $item->setBlub(uniqid());
            $item->setBlub2(uniqid());
            $item->setBlubLocalized(uniqid());
            $item->setKey($i.'-'.uniqid());
            $item->setParent(Service::createFolderByPath('/blub-'.$itemsToCreate.'/'.$i % 100));
            $item->save();

            if ($i % 100 === 0) {
                Db::get()->commit();
                Db::get()->beginTransaction();
            }

            $progress->advance();
        }

        Db::get()->commit();
    }

}

TL;DR

Pimcore uses a lot of memory on persisting a lot of items. I am not sure why and how to optimize. But importing thousands of items in one import is not possible at the moment.

The text was updated successfully, but these errors were encountered:

brusch · 2018-07-19T08:50:37Z

Have u tried \Pimcore::collectGarbage() ?

brusch · 2018-07-19T08:54:16Z

For example:

<?php

namespace AppBundle\Command;

use Pimcore\Model\DataObject\Service;
use Pimcore\Model\DataObject\Test;
use Symfony\Bundle\FrameworkBundle\Command\ContainerAwareCommand;
use Symfony\Component\Console\Helper\ProgressBar;
use Symfony\Component\Console\Input\InputArgument;
use Symfony\Component\Console\Input\InputInterface;
use Symfony\Component\Console\Output\OutputInterface;

class AppTestCommand extends ContainerAwareCommand
{
    protected function configure()
    {
        $this
            ->setName('app:test')
            ->addArgument('itemsToCreate', InputArgument::REQUIRED);
    }

    protected function execute(InputInterface $input, OutputInterface $output)
    {
        $itemsToCreate = $input->getArgument('itemsToCreate');

        $progress = new ProgressBar($output, $itemsToCreate);
        $progress->setFormat(
            '%current%/%max% [%bar%] %percent:3s%% (%elapsed:6s%/%estimated:-6s%) %memory:6s%: %message%'
        );

        for ($i = 0; $i < $itemsToCreate; $i++) {
            $progress->setMessage('Create Item with Number '.$i);

            $item = new Test();
            $item->setBlub(uniqid());
            $item->setBlub2(uniqid());
            $item->setBlubLocalized(uniqid());
            $item->setKey($i.'-'.uniqid());
            $item->setParent(Service::createFolderByPath('/blub-'.$itemsToCreate.'/'.$i % 100));
            $item->save();

            if ($i % 10 === 0) {
                \Pimcore::collectGarbage();
            }

            $progress->advance();
        }
    }

}

dpfaffenbauer · 2018-07-19T08:55:45Z

Yes, several times, here are the test results, but only with 1k items, as waiting another 20minutes is quite boring :P

With garbage collect:

Amount	Time	Start Memory Usage	End Memory Usage	Blackfire Report
1000	~2min	16mib	88mib	https://blackfire.io/profiles/8f572586-412d-43ab-a953-7fbfd8ab1c55/graph

Collect Garbage actually consumes 3.33mb of memory as well (Pimcore::collectGarbage memory)

command used:

<?php

namespace AppBundle\Command;

use Pimcore\Db;
use Pimcore\Model\DataObject\Service;
use Pimcore\Model\DataObject\Test;
use Symfony\Bundle\FrameworkBundle\Command\ContainerAwareCommand;
use Symfony\Component\Console\Helper\ProgressBar;
use Symfony\Component\Console\Input\InputArgument;
use Symfony\Component\Console\Input\InputInterface;
use Symfony\Component\Console\Output\OutputInterface;

class AppTestCommand extends ContainerAwareCommand
{
    protected function configure()
    {
        $this
            ->setName('app:test')
            ->addArgument('itemsToCreate', InputArgument::REQUIRED);
    }

    protected function execute(InputInterface $input, OutputInterface $output)
    {
        $itemsToCreate = $input->getArgument('itemsToCreate');

        $progress = new ProgressBar($output, $itemsToCreate);
        $progress->setFormat(
            '%current%/%max% [%bar%] %percent:3s%% (%elapsed:6s%/%estimated:-6s%) %memory:6s%: %message%'
        );

        for ($i = 0; $i < $itemsToCreate; $i++) {
            $progress->setMessage('Create Item with Number '.$i);

            $item = new Test();
            $item->setBlub(uniqid());
            $item->setBlub2(uniqid());
            $item->setBlubLocalized(uniqid());
            $item->setKey($i.'-'.uniqid());
            $item->setParent(Service::createFolderByPath('/blub-'.$itemsToCreate.'/'.$i % 100));
            $item->save();

            if ($i % 100 === 0) {
                \Pimcore::collectGarbage();
            }

            $progress->advance();
        }
    }

}

dpfaffenbauer · 2018-07-19T08:58:56Z

I also have to say that it runs on DEV Environment, on an PROD it doesn't consume as much, but increases in a same rate.

brusch · 2018-07-23T10:20:43Z

@dvesh3 can you try to reproduce the problem? Thanks!

AlternateIf · 2018-07-31T13:26:26Z

@brusch
i experienced a very similar behavior importing multiple items from another interface. A big chunk may be from buffered loggers but i think i also experienced an increase with loggers turned off

I ended up disabling the profiler (injecting it into my command) and unsetting the sql logger for my SyncCommand to lower the ram usage

if ($this->profiler !== null) { $this->profiler->disable(); }

Db::getConnection()->getConfiguration()->setSQLLogger(null);

dpfaffenbauer · 2018-07-31T15:14:39Z

made some more tests:

The biggest improvement is disabling the SQL Logger

Tests with profiler disabled and no SQL Logger

Amount	Time	Start Memory Usage	End Memory Usage	Blackfire Report
1000	~1.5min	16mib	40mib	https://blackfire.io/profiles/393b456d-734e-4ad0-b702-e16bd441d8c3/graph
10000	~27min	16mib	218mib	https://blackfire.io/profiles/fcded091-5ab8-418a-9d4a-8129ea2296e0/graph

Tests with profiler disabled and no SQL Logger and Versions disabled

Amount	Time	Start Memory Usage	End Memory Usage	Blackfire Report
1000	~1.3min	16mib	38mib	https://blackfire.io/profiles/ffc99435-3277-47bb-8666-c75f12dc4558/graph

Tests with profiler disabled and no SQL Logger and Versions disabled and Cache disabled

Amount	Time	Start Memory Usage	End Memory Usage	Blackfire Report
1000	~1.25min	16mib	36mib	https://blackfire.io/profiles/df9cb317-d379-4f83-8f23-bb0b8b3493ae/graph

command used:

<?php

namespace AppBundle\Command;

use Pimcore\Cache;
use Pimcore\Db;
use Pimcore\Model\DataObject\Service;
use Pimcore\Model\DataObject\Test;
use Pimcore\Model\Version;
use Symfony\Bundle\FrameworkBundle\Command\ContainerAwareCommand;
use Symfony\Component\Console\Helper\ProgressBar;
use Symfony\Component\Console\Input\InputArgument;
use Symfony\Component\Console\Input\InputInterface;
use Symfony\Component\Console\Output\OutputInterface;

class AppTestCommand extends ContainerAwareCommand
{
    protected function configure()
    {
        $this
            ->setName('app:test')
            ->addArgument('itemsToCreate', InputArgument::REQUIRED);
    }

    protected function execute(InputInterface $input, OutputInterface $output)
    {
        $this->getContainer()->get('profiler')->disable();
        Db::getConnection()->getConfiguration()->setSQLLogger(null);
//      Version::disable(); //Second run
//      Cache::disable(); //Third run

        $itemsToCreate = $input->getArgument('itemsToCreate');

        $progress = new ProgressBar($output, $itemsToCreate);
        $progress->setFormat(
            '%current%/%max% [%bar%] %percent:3s%% (%elapsed:6s%/%estimated:-6s%) %memory:6s%: %message%'
        );

        for ($i = 0; $i < $itemsToCreate; $i++) {
            $progress->setMessage('Create Item with Number '.$i);

            $item = new Test();
            $item->setBlub(uniqid());
            $item->setBlub2(uniqid());
            $item->setBlubLocalized(uniqid());
            $item->setKey($i.'-'.uniqid());
            $item->setParent(Service::createFolderByPath('/blub-'.$itemsToCreate.'/'.$i % 100));
            $item->save();

            $progress->advance();
        }
    }

}

dpfaffenbauer · 2018-07-31T15:39:31Z

Some tests with ENV=prod

Amount	Time	Start Memory Usage	End Memory Usage	Blackfire Report
1000	47s	14mib	28mib	https://blackfire.io/profiles/396ea4b9-0e6a-444c-a971-dee6afc2a67a/graph

Tests with profiler disabled and no SQL Logger and Versions disabled and Cache disabled

Amount	Time	Start Memory Usage	End Memory Usage	Blackfire Report
1000	29s	14mib	24mib	https://blackfire.io/profiles/76bcda0e-e92c-4175-a392-9c82e3e57232/graph

dvesh3 · 2018-08-03T04:40:13Z

I used @dpfaffenbauer Import command to import 20k objects in DEV mode

<?php

namespace AppBundle\Command;

use Pimcore\Model\DataObject\Service;
use Pimcore\Model\DataObject\Test;
use Symfony\Bundle\FrameworkBundle\Command\ContainerAwareCommand;
use Symfony\Component\Console\Helper\ProgressBar;
use Symfony\Component\Console\Input\InputArgument;
use Symfony\Component\Console\Input\InputInterface;
use Symfony\Component\Console\Output\OutputInterface;

class AppTestCommand extends ContainerAwareCommand
{
    protected function configure()
    {
        $this
            ->setName('app:test')
            ->addArgument('itemsToCreate', InputArgument::REQUIRED);
    }

    protected function execute(InputInterface $input, OutputInterface $output)
    {
        $itemsToCreate = $input->getArgument('itemsToCreate');

        $progress = new ProgressBar($output, $itemsToCreate);
        $progress->setFormat(
            '%current%/%max% [%bar%] %percent:3s%% (%elapsed:6s%/%estimated:-6s%) %memory:6s%: %message%'
        );

        for ($i = 0; $i < $itemsToCreate; $i++) {
            $progress->setMessage('Create Item with Number '.$i);

            $item = new Test();
            $item->setBlub(uniqid());
            $item->setBlub2(uniqid());
            $item->setBlubLocalized(uniqid());
            $item->setKey($i.'-'.uniqid());
            $item->setParent(Service::createFolderByPath('/blub-'.$itemsToCreate.'/'.$i % 100));
            $item->save();

            if ($i % 10 === 0) {
                \Pimcore::collectGarbage();
            }

            $progress->advance();
        }
    }

}

without tweaking any configuration and using collectGarbage()

Amount	Time	Start Memory Usage	End Memory Usage
20000	25mins	20Mib	1.5 Gib

after disabling Symfony doctrine "logging" and "profiling"

doctrine:
    dbal:
        default_connection: default
        connections:
            default:
                ........
                logging: false
                profiling: false

Amount	Time	Start Memory Usage	End Memory Usage
20000	21mins	20Mib	286.3 Mib

@brusch I think we can introduce these configurations in System settings. your suggestion?

brusch · 2018-08-03T05:37:01Z

@dvesh3 thank you very much for sharing your findings!
Yes, I think it makes sense to use this as the default and override them for the dev environment, I've pushed this changes already: 142067d

Any more potential to optimize the memory consumption? Monolog maybe?

Thanks!

dpfaffenbauer · 2018-08-03T06:42:39Z

Thats a huge improvement there. Another one would be to see where those 260MB of ram go. I mean, the operation is not too heavy to loose a lot of ram there.

AlternateIf · 2018-08-03T07:07:54Z

@dpfaffenbauer if you want to keep on using the dev env (e.g. for developing) you can run your commands with the '--no-debug' flag. This should actually disable the sql-logger

Since logging and profiling is still enabled for dev in 142067d

dpfaffenbauer · 2018-08-03T07:32:59Z

@AlternateIf Yeah I know, but thanks.

@brusch @dvesh3 I am currently preparing a test with Doctrine ORM as a comparison. I am importing 100000 datasets with 3 entities (to sort of simulate Pimcore a bit) and it only takes 3 mins and 18mb of RAM to do that without any increase at all.

https://blackfire.io/profiles/69f22bfd-2427-4204-baa7-ba4f823cd91f/graph

Importing a million sets would only take about 20 mins. And that is with blackfire agent running. Without it would only take about 10 mins.

dpfaffenbauer · 2018-08-03T10:43:01Z

@brusch One solution would be to calculate change-sets and only persist actual changes from the object, like doctrine ORM does.

dvesh3 · 2018-08-03T13:32:03Z

@brusch disabling Monolog doesn't help. @dpfaffenbauer i think this will require huge efforts as changeset calculation works on managed entities.

brusch · 2018-08-03T14:50:27Z

@dvesh3 thanks! should we try to find out by profiling the scripts above (Xdebug / Blackfire) ?

dpfaffenbauer · 2018-08-03T15:06:28Z

@dvesh3 Sure, but the results would be amazing. I think that Pimcore's DataObject could work pretty well with Doctrine. But probably only ORM.NEXT and not 2.5.

Anyways, I bought a Blackfire subscription to get more insights. Here is the graph with 1k items (same command): https://blackfire.io/profiles/08af9edc-72b9-4129-a824-e783a427045f/graph

Pimcore executes 24 216 queries inserting 1000 items and thats one of the memory problems.

dpfaffenbauer · 2018-08-03T16:51:26Z

Following changes improved memory usage a lot (like 50%):

https://github.com/pimcore/pimcore/blob/master/models/DataObject/Concrete/Dao.php#L223

change to

if ($isUpdate) {
            $this->db->update('object_store_' . $this->model->getClassId(), $data, ['oo_id' => $this->model->getId()]);
        }
        else {
            $this->db->insert('object_store_' . $this->model->getClassId(), $data);
        }

https://github.com/pimcore/pimcore/blob/master/models/DataObject/Concrete/Dao.php#L326

change to

$this->db->update('object_query_' . $this->model->getClassId(), $data, ['oo_id' => $this->model->getId()]);

It seems like insertOrUpdate uses a lot of memory.

https://github.com/pimcore/pimcore/blob/master/lib/Cache/Core/CoreHandler.php#L747

change to

    protected function addClearedTags($tags)
    {
        if (!is_array($tags)) {
            $tags = [$tags];
        }

        foreach ($tags as $tag) {
            if (!in_array($tag, $this->clearedTags)) {
                $this->clearedTags[] = $tag;
            }
        }

        return $this;
    }

array_unique takes a lot of memory with large arrays as well.

In general it seems that the current Cache implementation with MySQL is super slow. CoreShop uses the Pimcore Cache for all the Doctrine stuff, and that is very slow. Getting data directly from DB and hydrate it again is way faster than the Cache implementation.

Anyways, these 3 optimization helped me to come from 200mb of memory down to 86mb. Which is a good improvement. I am not using the Test Command here, I am using a more complex DataObject (Product from CoreShop) with some custom Editables and Object Indices.

dvesh3 · 2018-08-07T14:39:30Z

5K Import with TestCommand

Dev mode (doctrine profiling and logging enabled)
Memory: 434Mb
Time: 8min44s
Report: https://blackfire.io/profiles/ea02ffa6-72b6-4d1c-9fb5-cc5d5b12383a/graph?settings%5BtabPane%5D=recommendations

Memory Utilization	Resource
132Mib	update in Pimcore\Model\DataObject\Concrete\Dao
207Mib	saveVersion in Pimcore\Model\DataObject\Test
29.2Mib	update in Pimcore\Model\DataObject\Test
65.8Mib	Others

Prod mode with Dominik changes
Memory: 81Mb
Time: 6mins 19s
Report: https://blackfire.io/profiles/8bab81d5-9082-4f5b-93c1-a8300c2ed827/graph

Prod mode without Dominik changes
Memory: 54.3Mb
Time: 8mins 2s
Report: https://blackfire.io/profiles/77b3a748-00ac-4393-9bda-caca7c1d5d8b/graph

Prod mode with version disabled
Memory: 53MB
Time: 4mins 48s
Report: https://blackfire.io/profiles/06b212d8-57c1-437d-9efe-223d3e07c93d/graph

@dpfaffenbauer i don't think insertOrUpdate really causing this problem

dpfaffenbauer · 2018-08-09T09:39:37Z

@dvesh3 hmmm.... odd, it made it better for me :D, did you try that with the simple class or a more complex one with many relations?

dvesh3 · 2018-08-09T10:18:38Z

@dpfaffenbauer yeah, even i was not sure so cleared cache and tried it multiple times with same TestCommand but didn't get better result.

dpfaffenbauer · 2018-08-09T11:59:46Z

@dvesh3 Can you rerun your tests with and without my changes but warmup the cache before running blackfire. In your test with my changes, the kernel was reinitialized and that took about 40 mb of ram. So these comparisons aren't really valuable.

RuudvdvNB · 2018-08-09T14:06:52Z

For what it's worth, we are experiencing the same performance issues but not with any import. When we clear the cache (without warm-up) and reload the website memory goes up to 2.4GB and CPU goes up to 200% after which the site gets killed by our server.

In the log i see lots of deadlocks on the cache table.

edit
In this project we make use of multi-site (8 different sites) all using assets. Images are rendered with thumbnails so it's making heavy use of creating images & cache.

dvesh3 · 2018-08-09T15:03:20Z

@dpfaffenbauer Ok so I reran the same script after cache warmup for 5k import, there is a slight difference with your changes. I'm not sure what made your result better with these changes.

Script	Memory	Time	Report
5k Import without your changes	54.2Mb	8min 10s	https://blackfire.io/profiles/0a1391cb-18bf-4bb4-8040-0819146ae5c4/graph
5k Import with your changes	53.3Mb	7min 38 s	https://blackfire.io/profiles/e9c5008e-388c-415d-8bd0-1f242e86c342/graph

dpfaffenbauer · 2018-08-09T19:17:02Z

ok, so my changes do affect performance ;)

I guess the reason it helps a lot for me is that I use very complex objects with a lot of relations and persisting one of those Objects actually executes a whole lot of things.

dpfaffenbauer · 2018-09-03T09:30:44Z

any updates on this?

brusch · 2018-09-03T09:33:13Z

We're still on it ...

head1328 · 2018-09-03T09:40:24Z

yesterday i had some problems with symfony and long running console tasks on production: if fingers_crossed monolog handler is active (and on production it is usually) you have to clear the buffer after each iteration. see this: symfony/monolog-bundle#118

and here is a good code repo with examples: https://github.com/LongRunning/LongRunning

brusch · 2018-09-03T13:31:49Z

Made some optimizations in af32940 @dvesh3 maybe you can run your tests against that ans see if it makes it any better?

Thanks!

dpfaffenbauer · 2018-09-04T12:33:43Z

@brusch just tested your changes, didn't improve anything at all, in fact the opposite happened, it now needs 5mb more memory. Unfortunately I can't share the blackfire results from that test. I will try to find some time for my public test.

brusch · 2018-09-04T13:24:00Z

@dpfaffenbauer 😲ok, that's surprising.
No idea how this changes can make it even worse, very interesting.

dpfaffenbauer · 2018-09-04T13:30:44Z

brusch · 2018-09-04T13:33:42Z

Yep, but that should be a one time thing since I've put the cleanup tasks into a dedicated service, but that's not a leak - so not really relevant, isn't it?

dpfaffenbauer · 2018-09-04T13:38:26Z

Maybe, not sure. But overall memory usage increased by around that much. I will of course do some more tests, also on a production server, cause those are more reliable than the ones in my dev env.

dvesh3 · 2018-09-04T14:03:37Z

@brusch it worked for me!!..almost 50% improvement in memory utilization however time increased by 20% so that's trade-off we should expect.

running 5k import script with latest changes brings following results:

Script	memory	time	records
New	25.9mb	7m 2s	5k
Old	51.7mb	6m 7s	5k

comparison: https://blackfire.io/profiles/compare/83eda6b2-89fc-4cca-a65e-34530ab4acc5...91d076aa-f6cd-4441-9d1e-f694003dd3ff/graph

dvesh3 · 2018-09-04T14:06:49Z

@dpfaffenbauer can you please try again with prod env? and share your findings. Thanks!

brusch · 2018-09-05T09:32:18Z

In my tests there is definitely an improvement, although \Doctrine\DBAL\Driver\PDOStatement::execute() is still leaking a bit.
Not sure what else we can do (as Pimcore) to improve this...

markus-moser · 2018-09-07T06:09:02Z

Not the biggest topic but why calls the runtime cache the container on each single call? Wouldn't it be better to hold the instance internally as soon as the final instance is created? It's no memory leak but all these container calls sum up to 3 percent of the total runtime in the blackfire report I looked at.

brusch · 2018-09-07T06:19:05Z

@markus-moser Good point, done, see: 48c0bc3

dpfaffenbauer · 2018-09-07T06:38:10Z

@markus-moser 👍

But I am quite surprised to be honest, cause the container already holds an instance and just gets it when it already has been created.... But anyhow, every optimization is good :)

dvesh3 · 2018-09-07T07:05:41Z

@brusch i don't feel good about latest optimization changes: 78.9mb for 5k import https://blackfire.io/profiles/08af69f9-f824-40fb-9c01-6568e1d65145/graph

brusch · 2018-09-07T07:09:30Z

@dvesh3 you mean that one? 48c0bc3
But can't find it somewhere in the profiling :(
If you take the optimizations away, is it back to normal?

brusch · 2018-09-07T07:16:41Z

@dvesh3 I see, you're right, let me check that. Thanks!

…em-leak issue introduced by 48c0bc3, see #3104

brusch · 2018-09-07T07:23:52Z

@dvesh3 fixed

…em-leak issue introduced by 48c0bc3, see #3104

brusch · 2018-09-12T15:39:47Z

Maybe that's the answer to our question?
doctrine/dbal#3047

brusch · 2018-11-27T14:16:57Z

I'm closing that for now, it really seems that the remaining issues are related to doctrine/dbal#3047

brusch assigned dvesh3 Jul 23, 2018

brusch added a commit that referenced this issue Sep 5, 2018

[Cache] Memory optimization in cache handler - see #3104

d771608

brusch added this to the 5.5.0 milestone Sep 6, 2018

brusch self-assigned this Sep 6, 2018

brusch added a commit that referenced this issue Sep 7, 2018

[Cache] Runtime cache optimizations, reduce container calls, see #3104

48c0bc3

brusch added a commit that referenced this issue Sep 7, 2018

[Cache] Runtime cache optimizations, reduce container calls - fixed m…

a48b521

…em-leak issue introduced by 48c0bc3, see #3104

brusch added a commit that referenced this issue Sep 7, 2018

[Cache] Runtime cache optimizations, reduce container calls - fixed m…

592731f

…em-leak issue introduced by 48c0bc3, see #3104

brusch removed this from the 5.5.0 milestone Oct 15, 2018

brusch closed this as completed Nov 27, 2018

dpfaffenbauer mentioned this issue Dec 2, 2019

Memory leaks on bulk objects save #5361

Closed

Pimcore Memory Leak on Importing thousands of Objects #3104

Pimcore Memory Leak on Importing thousands of Objects #3104

Comments

dpfaffenbauer commented Jul 19, 2018

TL;DR

brusch commented Jul 19, 2018

brusch commented Jul 19, 2018

dpfaffenbauer commented Jul 19, 2018

dpfaffenbauer commented Jul 19, 2018

brusch commented Jul 23, 2018

AlternateIf commented Jul 31, 2018

dpfaffenbauer commented Jul 31, 2018 • edited Loading

dpfaffenbauer commented Jul 31, 2018

dvesh3 commented Aug 3, 2018

brusch commented Aug 3, 2018

dpfaffenbauer commented Aug 3, 2018

AlternateIf commented Aug 3, 2018 • edited Loading

dpfaffenbauer commented Aug 3, 2018

dpfaffenbauer commented Aug 3, 2018

dvesh3 commented Aug 3, 2018

brusch commented Aug 3, 2018

dpfaffenbauer commented Aug 3, 2018

dpfaffenbauer commented Aug 3, 2018

dvesh3 commented Aug 7, 2018 • edited Loading

dpfaffenbauer commented Aug 9, 2018

dvesh3 commented Aug 9, 2018

dpfaffenbauer commented Aug 9, 2018

RuudvdvNB commented Aug 9, 2018 • edited Loading

dvesh3 commented Aug 9, 2018

dpfaffenbauer commented Aug 9, 2018

dpfaffenbauer commented Sep 3, 2018

brusch commented Sep 3, 2018

head1328 commented Sep 3, 2018

brusch commented Sep 3, 2018

dpfaffenbauer commented Sep 4, 2018

brusch commented Sep 4, 2018

dpfaffenbauer commented Sep 4, 2018

brusch commented Sep 4, 2018 • edited Loading

dpfaffenbauer commented Sep 4, 2018

dvesh3 commented Sep 4, 2018 • edited Loading

dvesh3 commented Sep 4, 2018

brusch commented Sep 5, 2018 • edited Loading

markus-moser commented Sep 7, 2018

brusch commented Sep 7, 2018

dpfaffenbauer commented Sep 7, 2018

dvesh3 commented Sep 7, 2018

brusch commented Sep 7, 2018

brusch commented Sep 7, 2018

brusch commented Sep 7, 2018

brusch commented Sep 12, 2018

brusch commented Nov 27, 2018

dpfaffenbauer commented Jul 31, 2018 •

edited

Loading

AlternateIf commented Aug 3, 2018 •

edited

Loading

dvesh3 commented Aug 7, 2018 •

edited

Loading

RuudvdvNB commented Aug 9, 2018 •

edited

Loading

brusch commented Sep 4, 2018 •

edited

Loading

dvesh3 commented Sep 4, 2018 •

edited

Loading

brusch commented Sep 5, 2018 •

edited

Loading