Commit flow rework and fixed collection logic #1155

malarzm · 2015-06-21T17:08:44Z

As per my comment in #1128 (comment) this PR takes care of following things:

Collections deletions/updates/snapshots should take place immediately after owning document query and before dispatching events (fixes @postUpdate events trigger at an unexpected moment when using EmbedMany #1145, fixes case when collection is not saved due to another document's LockException) (collections modified in postUpdate are no longer saved to database in same commit() which may be a BC break but is correct from our point of view, since postUpdate was meant to be triggered after updating document in database)
Simplify handling atomic collections by moving relevant logic to PersistenceBuilder (thing I've mentioned in Fix updating of embedded documents if their atomic collection is not dirty #1154)
If it's possible include recursively collections during document's insert
Fix case when deeply nested collections weren't saved properly by recursively including collections if set or setArray strategies are used (fixes Added failing test case for nested collection update #1104)
Possible superflous queries updating/deleting collections of to be deleted document are no longer issued
Move tests for atomic collections from Ticket folder to one place, ~~increase number of tests for atomic strategies by doubling existing tests for collections~~ (which will be done after review to not blur the diff) EDIT: looking at current tests they seem to be enough combined with how atomic strategies are handled now

This PR doesn't fix #1046 but makes it bloody easy to implement, all that is needed is to pass current document's version to CollectionPersister and take it into account in update query. I can include this here if we decide it's good idea or leave it for another PR.

blockjon · 2015-06-22T10:08:06Z

Something is wrong with this PR but I'm having trouble coming up with a functional test to exhibit the behavior.

In my application flow, there is a document with two strategy="set" EmbedMany elements. If I use the mongodb-odm master line, adding to these two EmbedMany elements works normally. When I use this branch, the elements are not appended to.

Although I can't figure out how to repro the problem in a functional test, I can show you the queries my app is firing when on the master line versus this branch:

mongodb-odm master line queries:

{"update":true,"query":{"_id":{"$id":"5587d3562e6be58a048b456a"},"revision":3},"newObj":{"$set":{"defaultChargeAccountId":"chargeAccount_stripe_card_user_741572580320049154_dWJ3LS2afUbo75Yg"},"$inc":{"revision":1}},"options":{"w":1},"db":"api-development","collection":"users"}
{"update":true,"query":{"_id":{"$id":"5587d3562e6be58a048b456a"}},"newObj":{"$set":{"chargeAccounts":[{"id":"chargeAccount_stripe_card_user_741572580320049154_dWJ3LS2afUbo75Yg","customerId":"cus_6TNsmtkHmzGlL5","provider":"stripe","method":"card","label":"Personal","labelType":"personal","createdTimestamp":1434964830,"source":"user","cardId":"card_6TNs3YPmftl4QF","ccLastFour":"9424","ccFingerprint":"dWJ3LS2afUbo75Yg","ccType":"Discover","ccDescription":"Consumer Credit - Rewards","ccBrand":"Discover","ccFunding":"credit","ccCountry":"US","validCvc":"pass","expMonth":12,"expYear":2022,"addressZip":"","billingType":"credit","ccIin":"601100"}]}},"options":{"w":1},"db":"api-development","collection":"users"}
{"update":true,"query":{"_id":{"$id":"5587d3562e6be58a048b456a"}},"newObj":{"$set":{"chargeAccountsHistory":[{"id":"chargeAccount_stripe_card_user_741572580320049154_dWJ3LS2afUbo75Yg","customerId":"cus_6TNsmtkHmzGlL5","provider":"stripe","method":"card","label":"Personal","labelType":"personal","createdTimestamp":1434964830,"source":"user","cardId":"card_6TNs3YPmftl4QF","ccLastFour":"9424","ccFingerprint":"dWJ3LS2afUbo75Yg","ccType":"Discover","ccDescription":"Consumer Credit - Rewards","ccBrand":"Discover","ccFunding":"credit","ccCountry":"US","validCvc":"pass","expMonth":12,"expYear":2022,"addressZip":"","billingType":"credit","ccIin":"601100"}]}},"options":{"w":1},"db":"api-development","collection":"users"}

the same code on this branch:

{"update":true,"query":{"_id":{"$id":"5587d3562e6be58a048b456a"},"revision":3},"newObj":{"$set":{"defaultChargeAccountId":"chargeAccount_stripe_card_user_741572580320049154_dWJ3LS2afUbo75Yg"},"$inc":{"revision":1}},"options":{"w":1},"db":"api-development","collection":"users"}

As you can see, the app seems to only update the "surface" of the record when I run my unit tests with this new doctrine-odm branch. In other words, I am missing the queries which should be updating the embedmany elements.

blockjon · 2015-06-22T10:48:23Z

I'm still trying to figure out how the UnitOfWork works.. but it seems like the chargeAccounts and chargeAccountsHistory elements are not considered to be scheduled collections when using this branch. That's just a guess.

malarzm · 2015-06-22T10:49:31Z

I'll try to hack a failing test case in few hours, at a glance it seems strange as set collections should still be handled in old manner... maybe it's related to some postUpdate flow which is modifying collections?

blockjon · 2015-06-22T10:54:32Z

It does not look like my app has any postUpdate events in the failure situation.

blockjon · 2015-06-22T11:13:09Z

@malarzm I'm out of ideas.. I tried to debug this for 3-4 hours.

malarzm · 2015-06-22T11:25:38Z

@blockjon could you maybe bisect this PR? Maybe this will give a hint

blockjon · 2015-06-22T11:26:49Z

Yes I can do this in about 10 hours since I'm about to go to sleep. :)

malarzm · 2015-06-22T11:28:38Z

@blockjon sure thing is, have a good one ;)

blockjon · 2015-06-22T18:29:08Z

@malarzm Bisect reveals that my problem is introduced in 3313e65

malarzm · 2015-06-22T19:35:32Z

@blockjon blind guess, but could you comment out unset($this->hasScheduledCollections[spl_object_hash($topmostOwner)][$oid]); from Uow::unscheduleCollection(Deletion|Update) and see if it fixes your issue?

blockjon · 2015-06-22T20:04:09Z

lib/Doctrine/ODM/MongoDB/UnitOfWork.php

+        $oid = spl_object_hash($coll);
+        $topmostOwner = $this->getOwningDocument($coll->getOwner());
+        unset($this->collectionDeletions[$oid]);
+        unset($this->hasScheduledCollections[spl_object_hash($topmostOwner)][$oid]);


@malarzm When I comment this line out, my problem goes away!!

Oh crap I think I know what I did wrong here... I'll push a fix in few minutes, hopefully it'll be that :)

malarzm · 2015-06-22T20:19:02Z

@blockjon can you check now?

blockjon · 2015-06-22T20:56:18Z

@malarzm works!

blockjon · 2015-06-22T20:56:42Z

I will now re-run all of my automatic and manual smoke tests...

malarzm · 2015-06-23T19:44:55Z

@blockjon is no message a good message? :)

blockjon · 2015-06-23T19:48:55Z

@malarzm I just heard back from QA.. they told me all tests pass. I am therefore interested in rolling this out to my production system. Is this in a fairly stable state now in your opinion?

malarzm · 2015-06-23T20:07:49Z

@blockjon I do hope it is, with new way of handling atomic collections it should be way more stable than original implementation (and once again thank you for all failing test cases for that stuff) :) For a head up I am going to merge #782 tomorrow (or tonight) and then rebase this PR (but I don't expect tests to start failing because of that)

jmikola · 2015-07-15T21:55:35Z

lib/Doctrine/ODM/MongoDB/Persisters/DocumentPersister.php

@@ -355,42 +363,29 @@ public function update($document, array $options = array())
        $id = $this->uow->getDocumentIdentifier($document);
        $update = $this->pb->prepareUpdateData($document);

-        if ( ! empty($update) || $this->uow->hasScheduledCollections($document)) {


Not sure why I didn't ask this earlier, but this seems like we might bump the version of a document even if there is no other update to apply. I realize you want to ensure handleCollections() is called at the bottom of this method, but those collection modifications will take place in separate write operations AFAIK.

Is the idea here that you want to verify the main document's write so we can throw a LockException before handling collections?

I believe there's still a concern about the collection updates not incorporating versioning/locking in their own write ops; however, that's not a new problem at all.

This is exactly the previous behaviour if only collection was to be updated (IIRC this was the very thing why hasScheduledCollections was introduced) - bump version and later update collections. From PR description:

This PR doesn't fix #1046 [investigate locking and collection updates] but makes it bloody easy to implement, all that is needed is to pass current document's version to CollectionPersister and take it into account in update query.

How it will be done is yet to be determined, however now we will have both document and its collections update in one place which is way better than before :)

jmikola · 2015-07-16T20:44:24Z

lib/Doctrine/ODM/MongoDB/UnitOfWork.php

@@ -894,12 +883,10 @@ private function computeAssociationChanges($parentDocument, array $assoc, $value

        if ($value instanceof PersistentCollection && $value->isDirty() && $assoc['isOwningSide']) {
            if ($topOrExistingDocument || strncmp($assoc['strategy'], 'set', 3) === 0) {
-                if ( ! $this->isCollectionScheduledForUpdate($value)) {


Offhand, should the conditional above be using CollectionHelper::usesSet() instead of strncmp(), or is it purposefully ignoring atomic sets?

For atomic sets $topOrExistingDocument will be true but changing this for consistency is good idea 👍

malarzm · 2015-07-16T21:55:18Z

@jmikola I'll run tomorrow over all comments once again and apply yours, thanks for all the input!

malarzm · 2015-07-18T12:23:03Z

Merged manually in 5b0ff1c

/cc @blockjon

jmikola · 2015-07-21T16:02:31Z

malarzm · 2015-07-21T16:04:18Z

beberlei · 2015-07-21T16:09:14Z

guilhermeblanco · 2015-07-21T16:10:52Z

Ocramius · 2015-07-21T16:11:42Z

asm89 · 2015-07-21T16:15:40Z

Ocramius · 2015-07-21T16:23:05Z

mrohnstock · 2015-07-21T16:24:14Z

alcaeus · 2015-07-21T17:56:27Z

blockjon · 2015-07-21T18:34:53Z

malarzm added the Task label Jun 21, 2015

malarzm added this to the 1.0.0 milestone Jun 21, 2015

This was referenced Jun 21, 2015

Create snapshot of collection after querying db #1128

Closed

@postUpdate events trigger at an unexpected moment when using EmbedMany #1145

Closed

Functional test: atomicSet: "Cannot update 'chapters.0.name' and 'chapters' at the same time" #1156

Closed

blockjon reviewed Jun 22, 2015
View reviewed changes

malarzm mentioned this pull request Jun 22, 2015

PostCommit event to allow a "second round" of database operations with clean UnitOfWork #1158

Closed

malarzm added 7 commits June 28, 2015 14:47

Scheduled collections arrays are now sets

fd360b8

Introduced CollectionHelper

4a49694

Group visited collections by topmost owner

1f952ab

Move collection updates to DocumentPersister

1f3440e

Include collections in insert query

de93908

Moved handling atomic collections upsert to PersistenceBuilder

bf570d5

Simplified handling atomic collection updates

c9c217b

Explicit use statement for CollectionHelper

ddf46df

malarzm mentioned this pull request Jul 7, 2015

Reindex numerically PersistentCollection after save #1130

Closed

jmikola reviewed Jul 15, 2015
View reviewed changes

malarzm mentioned this pull request Jul 16, 2015

Can't flush only specific documents #1149

Closed

jmikola reviewed Jul 16, 2015
View reviewed changes

jmikola mentioned this pull request Jul 16, 2015

Added failing test case for nested collection update #1104

Closed

malarzm added 2 commits July 17, 2015 17:47

Rewording comments, minor changes after review

4f2713f

Moved away tests for atomic strategies from Tickets

964af17

malarzm added a commit that referenced this pull request Jul 18, 2015

Merge pull request #1155

5b0ff1c

malarzm closed this Jul 18, 2015

malarzm deleted the commit-flow-rework branch November 9, 2015 20:01

Commit flow rework and fixed collection logic #1155

Commit flow rework and fixed collection logic #1155

Conversation

malarzm commented Jun 21, 2015

blockjon commented Jun 22, 2015

blockjon commented Jun 22, 2015

malarzm commented Jun 22, 2015

blockjon commented Jun 22, 2015

blockjon commented Jun 22, 2015

malarzm commented Jun 22, 2015

blockjon commented Jun 22, 2015

malarzm commented Jun 22, 2015

blockjon commented Jun 22, 2015

malarzm commented Jun 22, 2015

blockjon Jun 22, 2015

Choose a reason for hiding this comment

malarzm Jun 22, 2015

Choose a reason for hiding this comment

malarzm commented Jun 22, 2015

blockjon commented Jun 22, 2015

blockjon commented Jun 22, 2015

malarzm commented Jun 23, 2015

blockjon commented Jun 23, 2015

malarzm commented Jun 23, 2015

jmikola Jul 15, 2015

Choose a reason for hiding this comment

malarzm Jul 15, 2015

Choose a reason for hiding this comment

jmikola Jul 16, 2015

Choose a reason for hiding this comment

jmikola Jul 16, 2015

Choose a reason for hiding this comment

malarzm Jul 16, 2015

Choose a reason for hiding this comment

malarzm commented Jul 16, 2015

malarzm commented Jul 18, 2015

jmikola commented Jul 21, 2015

malarzm commented Jul 21, 2015

beberlei commented Jul 21, 2015

guilhermeblanco commented Jul 21, 2015

Ocramius commented Jul 21, 2015

asm89 commented Jul 21, 2015

Ocramius commented Jul 21, 2015

mrohnstock commented Jul 21, 2015

alcaeus commented Jul 21, 2015

blockjon commented Jul 21, 2015