Skip to content
A set of utilities to operate with Doctrine ORM's batch processing techniques
PHP Shell
Branch: master
Clone or download
PReimers Merge pull request #45 from Ocramius/dependabot/composer/doctrine/com…
…mon-tw-2.11.0

Update doctrine/common requirement from ^2.10.0 to ^2.11.0
Latest commit 6fe585e Oct 2, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
examples Do not wrap result in array Nov 4, 2018
src/DoctrineBatchUtils/BatchProcessing feat: raise the php version to php 7.2 Mar 11, 2019
test/DoctrineBatchUtilsTest/BatchProcessing Fix static code analysis Mar 11, 2019
.gitignore Fix static code analysis Mar 11, 2019
.travis.install.sh
.travis.yml Enable BC checks Mar 11, 2019
LICENSE Adding license file Oct 9, 2015
README.md Add note about maintainer Mar 3, 2019
UPGRADE.md Enforce BC compatibility Mar 11, 2019
composer.json Merge pull request #45 from Ocramius/dependabot/composer/doctrine/com… Oct 1, 2019
infection.json.dist Fix EOL EOF Mar 11, 2019
phpcs.xml.dist Add doctrine/coding-standard Mar 4, 2019
phpstan.neon Add phpstan/phpstan Mar 4, 2019
phpunit.xml.dist Upgraded PHPUnit config to good defaults Feb 5, 2018
psalm.xml Add vimeo/psalm Mar 4, 2019

README.md

DoctrineBatchUtils

This repository attempts to ease the pain of dealing with batch-processing in the context of Doctrine ORM transactions.

This repository is maintained by Patrick Reimers (PReimers).

License Current release Travis-CI build status

Installation

Supported installation method is via Composer:

composer require ocramius/doctrine-batch-utils

Current features

As it stands, the only implemented utility in this repository is an IteratorAggregate that wraps around a DB transaction and calls ObjectManager#flush() and ObjectManager#clear() on the given EntityManager.

Example (array iteration)

It can be used as following:

use DoctrineBatchUtils\BatchProcessing\SimpleBatchIteratorAggregate;

$object1  = $entityManager->find('Foo', 1);
$object2  = $entityManager->find('Bar', 2);

$iterable = SimpleBatchIteratorAggregate::fromArrayResult(
    [$object1, $object2], // items to iterate
    $entityManager,       // the entity manager to operate on
    100                   // items to traverse before flushing/clearing
);

foreach ($iterable as $record) {
    // operate on records here
}
$record freshness

Please note that the $record inside the loop will always be "fresh" (managed state), as the iterator re-fetches it on its own: this prevents you from having to manually call ObjectManager#find() on your own for every iteration.

Automatic flushing/clearing

In this example, the EntityManager will be flushed and cleared only once, but if there were more than 100 records, then it would flush (and clear) twice or more.

Example (query/iterators)

The previous example is still not memory efficient, as we are operating on a pre-loaded array of objects loaded by the ORM.

We can use queries instead:

use DoctrineBatchUtils\BatchProcessing\SimpleBatchIteratorAggregate;

$iterable = SimpleBatchIteratorAggregate::fromQuery(
    $entityManager->createQuery('SELECT f FROM Files f'),
    100 // flush/clear after 100 iterations
);

foreach ($iterable as $record) {
    // operate on records here
}

Or our own iterator/generator:

use DoctrineBatchUtils\BatchProcessing\SimpleBatchIteratorAggregate;

// This is where you'd persist/create/load your entities (a lot of them!)
$results = function () {
    for ($i = 0; $i < 100000000; $i += 1) {
        yield new MyEntity($i); // note: identifier must exist in the DB
    }
};
 
$iterable = SimpleBatchIteratorAggregate::fromTraversableResult(
    $results(),
    $entityManager,
    100 // flush/clear after 100 iterations
);

foreach ($iterable as $record) {
    // operate on records here
}

// eventually after all records have been processed, the assembled transaction will be committed to the database

Both of these approaches are much more memory efficient.

You can’t perform that action at this time.