MongoDB as EventStore #151

jaymecd · 2015-04-08T08:13:23Z

Introducing support of MongoDB as EventStore.

Features included in this PR:

read queries against primary nodes for strong data consistency
read committed isolation level to avoid reading partial event commits
test cases adopted for MongoDB

wjzijderveld · 2015-04-11T18:31:25Z

src/Broadway/EventStore/MongoDBEventStore.php

+
+        if (count($events) == 1) {
+            try {
+                $this->saveEvent(reset($events));


Is there a specific reason you choose to do this separately if there is exactly 1 event? I'm not really seeing the benefit.

Single document insert does not require 2pc. It's done to save number of requests.

wjzijderveld · 2015-04-11T19:11:15Z

Thanks for the PR, really appreciate it!

jaymecd · 2015-04-11T21:32:09Z

@wjzijderveld updated, but scrutinizer failed by some unexpected reason.
Any idea? I've contacted support, waiting.

wjzijderveld · 2015-04-12T06:08:19Z

I just restarted The scrutrnizer job, they fsil from time to time.

I'll look at you other comments later :-)

wjzijderveld · 2015-05-04T08:06:53Z

src/Broadway/EventStore/MongoDBEventStore.php

+    }
+
+    /**
+     * {@inheritdoc}


This method isn't in the interface, no doc-block required here.

- read against primary node - read committed isolation level

francescotrucchia · 2015-08-06T15:59:50Z

👍

wjzijderveld · 2015-08-07T20:05:25Z

@qandidate-labs/broadway I'm fine with merging this as is (can optionally fix the exception assignment separate). What do others think?

@jaymecd Have you been running this in production? If not, maybe we should add a note in the README that this is experimental?

jaymecd · 2015-08-07T20:09:52Z

@wjzijderveld nope, not yet, so experimental would be important.

prolic · 2015-08-07T22:20:53Z

src/Broadway/EventStore/MongoDBEventStore.php

+     */
+    public function configureCollection()
+    {
+        $this->eventCollection->ensureIndex(array(


ensureIndex is deprecated. use createIndex instead.

sorry, doctrine mongodb uses still the ensureIndex method ;)

jaymecd · 2015-08-17T12:56:07Z

@jmikola tnx for tips, I review code a bit later.

jmikola · 2015-08-17T12:59:03Z

@jaymecd: No problem. @asm89 just sent me this PR over IRC and invited me to take a look :)

danizord · 2015-09-17T18:40:51Z

Any news on this one?

prolic · 2015-09-20T13:47:23Z

MongoDB has no transaction support. Using the $isolated operator you can achieve atomic updates on a single collection, but not on multiple ones. Additionally the $isolated operator does not work in a sharded cluster environment. That said, it's not a good idea to run this in a very high performant situation with sharding enabled.
You can check out our implementation of the mongo-db-adapter for prooph/event-store, to see how this is working with manual transaction support. The PR to address the $isolated operator and changing README notes is here: prooph/event-store-mongodb-adapter#27.

jaymecd · 2015-10-07T09:21:33Z

Hey guys, I spent last weeks in hospital. Continue with PR in next days. Tnx for understanding.

wjzijderveld · 2015-10-19T15:26:53Z

@jaymecd No problem, hope you are okay!

If you want any help on it, let me know!

jaymecd · 2015-11-20T10:21:24Z

@wjzijderveld, I've made a deep dive into the problems and issues noted by @jmikola and @prolic and have doubts if solution should be based on:

mongo extension (legacy)
- recently marked as deprecated
- used by many
mongodb extension
- supersedes mongo extension
- supports both php and hhvm
- has own userland library mongodb/mongo-php-library
- NB! do not support legacy-based integration, eg. Doctrine MongoDB, due to the different API

So, I'm asking for your opinion on this issue before continue. Personally I'd go with mongodb extension/library and then, if it's required, with legacy extension support as separate storage class MongoDBLegacyEventStore.

wjzijderveld · 2015-11-20T12:25:03Z

@jaymecd Thanks for diving into it!

Hmm, tricky choice there. But I think the better way would be to go with the new extension. Compatibility with HHVM is something I would definitely like to support :-)
Downsides: we use the deprecated mongo extension (with Doctrine) for Saga states at the moment. So we probable should do something about that as well.

@qandidate-labs/broadway What do you think?

codeliner · 2015-11-20T16:56:12Z

Also the legacy driver will not be updated to support php7. No HHVM support ok, but PHP7 should not be a question ;) If you also want to have legacy version you can wrap the 1.0 version of prooph/event-store-mongodb-adapter in a class implementing Broadway event store. Maybe faster than writing a legacy adapter yourself.
Our 2.0 version will use the new extension.

jaymecd · 2015-11-20T20:32:47Z

@codeliner, I don't think wrapping prooph implementation would be a good idea, as it brings another level to diamond dependencies + prooph/common: one ES implementation partially includes another one - that's more than strange.

jaymecd · 2015-11-20T20:38:09Z

@codeliner @jmikola - $isolated operator does not work in a sharded cluster.

Is there any way to have same atomicity within sharded setup? As event store data could consume all disk space on single shard, isn't it?

codeliner · 2015-11-20T21:21:26Z

@jaymecd : @prolic is our mongoDB specialist. As far as I know there is no way. We are looking for alternatives, too. Depending on your domain contexts one shard per context is maybe enough. Also when using snapshots in combination with "archiving" old events you may not run out of disk space. So mongoDB is still a valid option.

one ES implementation partially includes another one

Well, Broadway provides only an ES interface and different independent ES implementations. prooph/event-store on the other hand uses different adapters. I did not say you should require prooph/event-store but only one of the available adapters. Our packages are stand-alone (btw. something that Broadway should consider, too). However, it was more a hint for people who may need to use the old extension but want to use other features of Broadway. I don't consider Broadway a competitor. Would like to use the Saga implementation but unfortunately it is not available as a stand-alone package and installing everything incl. the Sf bundle, aggregate root classes, etc. is really not an option.

prolic · 2015-11-20T22:50:24Z

Running out of disk-space? Really? I can store all your events on my micro-sd card! Unless you are not twitter and have multiple thousands events per second, you never run out of disk space. Even if so, you should upgrade your hardware and buy a larger hard disk.

For $isolated: There is no other solution, you can't run mongodb event store in a sharded cluter environment. If you need more speed, consider using postgres.

sstok · 2015-11-21T16:09:21Z

sharded cluter epic typo 😂

If you need more speed, consider using postgres.

Here 👍

jaymecd · 2015-11-24T11:04:10Z

In order to save time I've started to update implementation (using hints above) with mongodb extension and it's library to have MongoDB as event store, as PHP7 & HHVM really worth it.

Konafets · 2016-02-01T08:18:57Z

Hi @jaymecd,

In order to save time I've started to update implementation (using hints above) with mongodb extension and it's library to have MongoDB as event store, as PHP7 & HHVM really worth it.

How is your progress on that?

wjzijderveld · 2016-02-01T21:39:24Z

I started to refactor the Saga\MongoDBRepository to use the new extension #236

jaymecd · 2016-02-02T12:15:51Z

@Konafets @wjzijderveld I didn't had a chance to continue on that, so far could not provide any ETA from my side.
I would highly appreciate if you could continue on his PR.

othillo · 2017-02-18T12:34:24Z

I'm closing this for now. Additional event stores are very welcome, though! We are actually splitting implementations to separate repositories at the moment. If someone wants to work on a new broadway/event-store-mongodb implementation, please ping us! /cc @broadway/broadway

Thank you for your effort!

othillo · 2017-02-22T18:46:15Z

work on this will continue in this repo: https://github.com/broadway/event-store-mongodb

jaymecd force-pushed the eventstore_mongodb branch from 8f32753 to 6c391ba Compare April 8, 2015 19:26

wjzijderveld reviewed Apr 11, 2015
View reviewed changes

jaymecd force-pushed the eventstore_mongodb branch 3 times, most recently from 31be3bf to 035bfe9 Compare April 11, 2015 21:24

wjzijderveld reviewed May 4, 2015
View reviewed changes

MongoDB as EventStore

c1731ef

- read against primary node - read committed isolation level

jaymecd force-pushed the eventstore_mongodb branch from 035bfe9 to c1731ef Compare May 4, 2015 08:30

prolic reviewed Aug 7, 2015
View reviewed changes

othillo closed this Feb 18, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MongoDB as EventStore #151

MongoDB as EventStore #151

jaymecd commented Apr 8, 2015

wjzijderveld Apr 11, 2015

jaymecd Apr 11, 2015

wjzijderveld commented Apr 11, 2015

jaymecd commented Apr 11, 2015

wjzijderveld commented Apr 12, 2015

wjzijderveld May 4, 2015

jaymecd May 4, 2015

francescotrucchia commented Aug 6, 2015

wjzijderveld commented Aug 7, 2015

jaymecd commented Aug 7, 2015

prolic Aug 7, 2015

prolic Aug 7, 2015

jaymecd commented Aug 17, 2015

jmikola commented Aug 17, 2015

danizord commented Sep 17, 2015

prolic commented Sep 20, 2015

jaymecd commented Oct 7, 2015 via email

wjzijderveld commented Oct 19, 2015

jaymecd commented Nov 20, 2015

wjzijderveld commented Nov 20, 2015

codeliner commented Nov 20, 2015

jaymecd commented Nov 20, 2015

jaymecd commented Nov 20, 2015

codeliner commented Nov 20, 2015

prolic commented Nov 20, 2015

sstok commented Nov 21, 2015

jaymecd commented Nov 24, 2015

Konafets commented Feb 1, 2016

wjzijderveld commented Feb 1, 2016

jaymecd commented Feb 2, 2016

othillo commented Feb 18, 2017

othillo commented Feb 22, 2017 •

edited

MongoDB as EventStore #151

MongoDB as EventStore #151

Conversation

jaymecd commented Apr 8, 2015

wjzijderveld Apr 11, 2015

Choose a reason for hiding this comment

jaymecd Apr 11, 2015

Choose a reason for hiding this comment

wjzijderveld commented Apr 11, 2015

jaymecd commented Apr 11, 2015

wjzijderveld commented Apr 12, 2015

wjzijderveld May 4, 2015

Choose a reason for hiding this comment

jaymecd May 4, 2015

Choose a reason for hiding this comment

francescotrucchia commented Aug 6, 2015

wjzijderveld commented Aug 7, 2015

jaymecd commented Aug 7, 2015

prolic Aug 7, 2015

Choose a reason for hiding this comment

prolic Aug 7, 2015

Choose a reason for hiding this comment

jaymecd commented Aug 17, 2015

jmikola commented Aug 17, 2015

danizord commented Sep 17, 2015

prolic commented Sep 20, 2015

jaymecd commented Oct 7, 2015 via email

wjzijderveld commented Oct 19, 2015

jaymecd commented Nov 20, 2015

wjzijderveld commented Nov 20, 2015

codeliner commented Nov 20, 2015

jaymecd commented Nov 20, 2015

jaymecd commented Nov 20, 2015

codeliner commented Nov 20, 2015

prolic commented Nov 20, 2015

sstok commented Nov 21, 2015

jaymecd commented Nov 24, 2015

Konafets commented Feb 1, 2016

wjzijderveld commented Feb 1, 2016

jaymecd commented Feb 2, 2016

othillo commented Feb 18, 2017

othillo commented Feb 22, 2017 • edited

othillo commented Feb 22, 2017 •

edited