Implement NoSQL Storage for ActiveRecord #11

Closed
cebe opened this Issue Mar 24, 2013 · 71 comments

Comments

Projects
None yet
@cebe
Member

cebe commented Mar 24, 2013

Will implement:

redis (versions >= 2.6) (pull request: #905 merged)

elastic search (pull request: #1295 merged)

mongodb (pull request: #1438 merged)

Sphinx (Pull Request merged)

http://sphinxsearch.com/

Other systems that may get implemented (suggestions welcome):

Other resources:

@ghost ghost assigned cebe Mar 24, 2013

@cebe

This comment has been minimized.

Show comment
Hide comment
@cebe

cebe Mar 24, 2013

Member

Btw the checkboxes in here are made like this: https://github.com/blog/1375-task-lists-in-gfm-issues-pulls-comments

Member

cebe commented Mar 24, 2013

Btw the checkboxes in here are made like this: https://github.com/blog/1375-task-lists-in-gfm-issues-pulls-comments

@oherych

This comment has been minimized.

Show comment
Hide comment
@oherych

oherych Mar 25, 2013

Most noSQL databases have connection via REST. Will be greate have one defined interface for all Rest client. This give us able have one selected (by developer) REST library for all needs. We just say with which library it should use

oherych commented Mar 25, 2013

Most noSQL databases have connection via REST. Will be greate have one defined interface for all Rest client. This give us able have one selected (by developer) REST library for all needs. We just say with which library it should use

@Aoseala

This comment has been minimized.

Show comment
Hide comment
@Aoseala

Aoseala Mar 25, 2013

Contributor

Personal think elasticsearch PHP client, Elastica is very good! this is his github repository URL: https://github.com/ruflin/Elastica

Contributor

Aoseala commented Mar 25, 2013

Personal think elasticsearch PHP client, Elastica is very good! this is his github repository URL: https://github.com/ruflin/Elastica

@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost Mar 25, 2013

You mentioned 2 of my yii extensions so I might share my thoughts/experiences on this issue here: The whole noSQL topic is far too complex to make a "one fits all" solution even possible (I've tried and failed :-) ). I wanted to make things easier with my ActiveResource extension but there are a lot of design flaws in there so it is a bad candidate for reviewing. My suggestion for Yii in this respect: Take a look at Guzzle which is a REST client framework. I takes care of the annoying parts (authentication, caching, asynchronous/parallel curl requests), is extensible via plugins and actively maintained. Yii could offer an interface on top of Guzzle so that there is a well defined API to develop against, maybe in combination with the repository pattern

ghost commented Mar 25, 2013

You mentioned 2 of my yii extensions so I might share my thoughts/experiences on this issue here: The whole noSQL topic is far too complex to make a "one fits all" solution even possible (I've tried and failed :-) ). I wanted to make things easier with my ActiveResource extension but there are a lot of design flaws in there so it is a bad candidate for reviewing. My suggestion for Yii in this respect: Take a look at Guzzle which is a REST client framework. I takes care of the annoying parts (authentication, caching, asynchronous/parallel curl requests), is extensible via plugins and actively maintained. Yii could offer an interface on top of Guzzle so that there is a well defined API to develop against, maybe in combination with the repository pattern

@DaSourcerer

This comment has been minimized.

Show comment
Hide comment
@DaSourcerer

DaSourcerer Mar 25, 2013

Contributor

As I see Guzzle being mentioned here: While Guzzle is indeed a formidable solution with facilities that make inclusion into Yii trivial, I'd like to advise caution. As mentioned before, it is not just a library but an entire framework. It would also introduce a hard dependency on libcurl (it is my impression that the original author simply wanted to spare himself from dealing with all the gory details of socket programming). Just saying 😏

Contributor

DaSourcerer commented Mar 25, 2013

As I see Guzzle being mentioned here: While Guzzle is indeed a formidable solution with facilities that make inclusion into Yii trivial, I'd like to advise caution. As mentioned before, it is not just a library but an entire framework. It would also introduce a hard dependency on libcurl (it is my impression that the original author simply wanted to spare himself from dealing with all the gory details of socket programming). Just saying 😏

@rawtaz

This comment has been minimized.

Show comment
Hide comment
@rawtaz

rawtaz Mar 25, 2013

Contributor

This is almost getting a bit OT, but I have to say: +1 for the repository pattern. Been missing it in Yii a lot of times.

Contributor

rawtaz commented Mar 25, 2013

This is almost getting a bit OT, but I have to say: +1 for the repository pattern. Been missing it in Yii a lot of times.

@PommeVerte

This comment has been minimized.

Show comment
Hide comment
@PommeVerte

PommeVerte Apr 15, 2013

Contributor

I would love to see HyperDex in this list. I know this is a bit of a longshot but it's got good performance, supports ACID transactions and has strong consistency guarantees and fault tolerance to boot.

It's outperformed MongoDB, Cassandra, and I believe Redis in the YCSB (despite better consistency guarantees and fault tolerance).
It's only downside really is that it's relatively young compared to other solutions.

Here are some links:

http://hyperdex.org/doc/
http://hyperdex.org/performance/
http://hyperdex.org/performance/details/ (more details regarding benchmark)
https://github.com/tritondigital/php-hyperdex

Alternativly I could settle with MongoDB even though I'm uncomfortable with some aspects of the data storage (or non-storage in this case)

Contributor

PommeVerte commented Apr 15, 2013

I would love to see HyperDex in this list. I know this is a bit of a longshot but it's got good performance, supports ACID transactions and has strong consistency guarantees and fault tolerance to boot.

It's outperformed MongoDB, Cassandra, and I believe Redis in the YCSB (despite better consistency guarantees and fault tolerance).
It's only downside really is that it's relatively young compared to other solutions.

Here are some links:

http://hyperdex.org/doc/
http://hyperdex.org/performance/
http://hyperdex.org/performance/details/ (more details regarding benchmark)
https://github.com/tritondigital/php-hyperdex

Alternativly I could settle with MongoDB even though I'm uncomfortable with some aspects of the data storage (or non-storage in this case)

@klimov-paul

This comment has been minimized.

Show comment
Hide comment
@klimov-paul

klimov-paul Apr 23, 2013

Member

Implement NoSQL Storage for ActiveRecord

I am not sure how exactly such implementation should look at the end.
I can not see an universal solution for this.
Different NOSQL storages use different approaches for the data storage and data structure.
There are several types:

  • key-value oriented (Redis)
  • document oriented (MongoDb)
  • graph oriented (Neo4j)

Key-value storages could match the original Yii model paradigm, which relies on architecture, when model has only (or at least mostly) scalar type attributes.

However document oriented storage does not fit the Yii model paradigm. In such storages (as MongoDb) the single storage entry is a huge nested JSON document, which can not be described by simple model with scalar attributes.

Graph oriented database (such as Neo4j) actually could fit the Yii model paradigm. However the search inside such storages is rather different from the common database. Here you are starting with some node and then go to the nodes related with it. Sometimes universal search criteria can not be composed for fetching particular node with the single query.
But the main problem with such storages as Neo4j is they do not support PHP directly and there are no PHP drivers for them. Such storages can be accessed only via REST API.

Member

klimov-paul commented Apr 23, 2013

Implement NoSQL Storage for ActiveRecord

I am not sure how exactly such implementation should look at the end.
I can not see an universal solution for this.
Different NOSQL storages use different approaches for the data storage and data structure.
There are several types:

  • key-value oriented (Redis)
  • document oriented (MongoDb)
  • graph oriented (Neo4j)

Key-value storages could match the original Yii model paradigm, which relies on architecture, when model has only (or at least mostly) scalar type attributes.

However document oriented storage does not fit the Yii model paradigm. In such storages (as MongoDb) the single storage entry is a huge nested JSON document, which can not be described by simple model with scalar attributes.

Graph oriented database (such as Neo4j) actually could fit the Yii model paradigm. However the search inside such storages is rather different from the common database. Here you are starting with some node and then go to the nodes related with it. Sometimes universal search criteria can not be composed for fetching particular node with the single query.
But the main problem with such storages as Neo4j is they do not support PHP directly and there are no PHP drivers for them. Such storages can be accessed only via REST API.

@cebe

This comment has been minimized.

Show comment
Hide comment
@cebe

cebe Apr 23, 2013

Member

@Haensel commented in #73 (comment):
I don't know if it is still the case but if you are still planing to implement a mapper for graph databases (Neo4j, OrientDB etc.) I would suggest you to take a look at Tinkerpop Blueprints (https://github.com/tinkerpop/blueprints/wiki). From the wiki:

"Blueprints is analogous to the JDBC, but for graph databases. As such, it provides a common set of interfaces to allow developers to plug-and-play their graph database backend. Moreover, software written atop Blueprints works over all Blueprints-enabled graph databases."

Neo4j and OrientDb both support Blueprints and are the most widely used graph dbs, so it might be worth a look

Member

cebe commented Apr 23, 2013

@Haensel commented in #73 (comment):
I don't know if it is still the case but if you are still planing to implement a mapper for graph databases (Neo4j, OrientDB etc.) I would suggest you to take a look at Tinkerpop Blueprints (https://github.com/tinkerpop/blueprints/wiki). From the wiki:

"Blueprints is analogous to the JDBC, but for graph databases. As such, it provides a common set of interfaces to allow developers to plug-and-play their graph database backend. Moreover, software written atop Blueprints works over all Blueprints-enabled graph databases."

Neo4j and OrientDb both support Blueprints and are the most widely used graph dbs, so it might be worth a look

@cebe

This comment has been minimized.

Show comment
Hide comment
@cebe

cebe May 4, 2013

Member

In case you wonder where to find this: I'm on the redis branch: https://github.com/yiisoft/yii2/tree/redis

Member

cebe commented May 4, 2013

In case you wonder where to find this: I'm on the redis branch: https://github.com/yiisoft/yii2/tree/redis

@PommeVerte

This comment has been minimized.

Show comment
Hide comment
@PommeVerte

PommeVerte May 12, 2013

Contributor

Is there an interface or required functions for Connection class? I would like to implement support for Rexster (as per the tinkerpop comment earlier). I've already made a php client for RexPro (which part of is very similar to the Yii Connection.php). I'd like to use it as a helper as a first step even if it's redundant (because I might move on to making a php extension to support rexPro.
I'm going to start now just trying to stick as much as possible to the current file/functionality.

Contributor

PommeVerte commented May 12, 2013

Is there an interface or required functions for Connection class? I would like to implement support for Rexster (as per the tinkerpop comment earlier). I've already made a php client for RexPro (which part of is very similar to the Yii Connection.php). I'd like to use it as a helper as a first step even if it's redundant (because I might move on to making a php extension to support rexPro.
I'm going to start now just trying to stick as much as possible to the current file/functionality.

@cebe

This comment has been minimized.

Show comment
Hide comment
@cebe

cebe May 12, 2013

Member

Is there an interface or required functions for Connection class?

Will be there but I am not finished with that yet. Will define the interface or base class, when we have more dbms implemented so I'd be very interested in seeing your class.

Member

cebe commented May 12, 2013

Is there an interface or required functions for Connection class?

Will be there but I am not finished with that yet. Will define the interface or base class, when we have more dbms implemented so I'd be very interested in seeing your class.

@PommeVerte

This comment has been minimized.

Show comment
Hide comment
@PommeVerte

PommeVerte May 14, 2013

Contributor

ok!
I haven't gone very far yet. I've used your Connection class as a basis and have made one for Rexter. I've left out Transactions and the concept of Isolation (or non-isolation in this case which allows for queries to use bindings set by previous queries) because it will probably need a bit of discussion .
I'll get onto looking into activeRecord and ActiveQuery classes next. Once I get ideas as to what to do about representation of Vertices vs Edges and routes and so forth. I'll probably post my suggestions. (I also should have a look at Neo4Yii to see how Hansael tackled it)

Contributor

PommeVerte commented May 14, 2013

ok!
I haven't gone very far yet. I've used your Connection class as a basis and have made one for Rexter. I've left out Transactions and the concept of Isolation (or non-isolation in this case which allows for queries to use bindings set by previous queries) because it will probably need a bit of discussion .
I'll get onto looking into activeRecord and ActiveQuery classes next. Once I get ideas as to what to do about representation of Vertices vs Edges and routes and so forth. I'll probably post my suggestions. (I also should have a look at Neo4Yii to see how Hansael tackled it)

@PommeVerte

This comment has been minimized.

Show comment
Hide comment
@PommeVerte

PommeVerte May 16, 2013

Contributor

ok having a look around. How would you suggest modifying __get() and __set() for shema-less db structures in activeRecord? This is kind of tricky as it'll catch all.

Contributor

PommeVerte commented May 16, 2013

ok having a look around. How would you suggest modifying __get() and __set() for shema-less db structures in activeRecord? This is kind of tricky as it'll catch all.

@PommeVerte

This comment has been minimized.

Show comment
Hide comment
@PommeVerte

PommeVerte Jun 2, 2013

Contributor

graph DB and gremlin discussion for graph implementation here : http://www.yiiframework.com/forum/index.php/topic/43863-graph-databases-gremlin-qquery-building/
I reckon I should have something to push online by the end of the week depending on how the discussion goes. a preview version if you will

Contributor

PommeVerte commented Jun 2, 2013

graph DB and gremlin discussion for graph implementation here : http://www.yiiframework.com/forum/index.php/topic/43863-graph-databases-gremlin-qquery-building/
I reckon I should have something to push online by the end of the week depending on how the discussion goes. a preview version if you will

@Sammaye

This comment has been minimized.

Show comment
Hide comment
@Sammaye

Sammaye Jun 19, 2013

I don't mean to plug my own content here but: https://github.com/Sammaye/MongoYii can handle MongoDB in a schemaless manner and I have the intention of converting it to Yii2 as soon as a stable interface for its conversion is made.

Maybe a stripped down version of something like that could be used for core?

Sammaye commented Jun 19, 2013

I don't mean to plug my own content here but: https://github.com/Sammaye/MongoYii can handle MongoDB in a schemaless manner and I have the intention of converting it to Yii2 as soon as a stable interface for its conversion is made.

Maybe a stripped down version of something like that could be used for core?

@cebe

This comment has been minimized.

Show comment
Hide comment
@cebe

cebe Jun 19, 2013

Member

It's nothing bad about showing your own content when you think it is good. Looks really interesting, will definitifely take a look at it. Thanks for sharing!

Member

cebe commented Jun 19, 2013

It's nothing bad about showing your own content when you think it is good. Looks really interesting, will definitifely take a look at it. Thanks for sharing!

@PommeVerte

This comment has been minimized.

Show comment
Hide comment
@PommeVerte

PommeVerte Aug 19, 2013

Contributor

I've been very busy recently but finally shifted towards making Yii independent libraries for graph DBs. Mostly speaking a DB client for rexster and an equivalent of a QueryBuilder for gremlin. Now I'm concentrating on integrating these into Yii2. I was wondering where the correct place for external libraries to reside was? I've been sticking them in the helpers dir for now even though that didn't seem correct. Any help would be appreciated.
Thanks

Contributor

PommeVerte commented Aug 19, 2013

I've been very busy recently but finally shifted towards making Yii independent libraries for graph DBs. Mostly speaking a DB client for rexster and an equivalent of a QueryBuilder for gremlin. Now I'm concentrating on integrating these into Yii2. I was wondering where the correct place for external libraries to reside was? I've been sticking them in the helpers dir for now even though that didn't seem correct. Any help would be appreciated.
Thanks

@Sammaye

This comment has been minimized.

Show comment
Hide comment
@Sammaye

Sammaye Aug 19, 2013

Normally, for better composer integration, many have taken to adding such things to a vendors folder. This pattern is, of course, also replicated in many languages including Java.

Sammaye commented Aug 19, 2013

Normally, for better composer integration, many have taken to adding such things to a vendors folder. This pattern is, of course, also replicated in many languages including Java.

@PommeVerte

This comment has been minimized.

Show comment
Hide comment
@PommeVerte

PommeVerte Aug 19, 2013

Contributor

Yeah, I couldn't seem to find one though. Doesn't matter, I'll place them there for now and move on. I'll move them around once I get a better idea.

Contributor

PommeVerte commented Aug 19, 2013

Yeah, I couldn't seem to find one though. Doesn't matter, I'll place them there for now and move on. I'll move them around once I get a better idea.

@jabbon

This comment has been minimized.

Show comment
Hide comment
@jabbon

jabbon Sep 6, 2013

Support for MongoDB +1 (ActiveRecord, Sessions)
Support for Redis +1 (Cache, Sessions)

jabbon commented Sep 6, 2013

Support for MongoDB +1 (ActiveRecord, Sessions)
Support for Redis +1 (Cache, Sessions)

@cebe

This comment has been minimized.

Show comment
Hide comment
@cebe

cebe Sep 7, 2013

Member

Support for Redis +1 (Cache, Sessions)

@jabbon this is already implemented and available on master branch.

Support for MongoDB +1 (ActiveRecord, Sessions)

does it really make sense to store Sessions on MongoDB?

Member

cebe commented Sep 7, 2013

Support for Redis +1 (Cache, Sessions)

@jabbon this is already implemented and available on master branch.

Support for MongoDB +1 (ActiveRecord, Sessions)

does it really make sense to store Sessions on MongoDB?

@Sammaye

This comment has been minimized.

Show comment
Hide comment
@Sammaye

Sammaye Sep 7, 2013

Yes, MongoDB, if done right, can store the session completely in working set, i.e. RAM just like redis

Sammaye commented Sep 7, 2013

Yes, MongoDB, if done right, can store the session completely in working set, i.e. RAM just like redis

@qiangxue

This comment has been minimized.

Show comment
Hide comment
@qiangxue

qiangxue Sep 14, 2013

Member

Suggest moving /db/redis to /redis because redis is relatively independent of the other classes under db.

Member

qiangxue commented Sep 14, 2013

Suggest moving /db/redis to /redis because redis is relatively independent of the other classes under db.

@cebe

This comment has been minimized.

Show comment
Hide comment
@cebe

cebe Sep 14, 2013

Member

The current classes are but the AR does extend from the db\ActiveRecord
I think we should make a decision when redis an elasticsearch are ready.

Member

cebe commented Sep 14, 2013

The current classes are but the AR does extend from the db\ActiveRecord
I think we should make a decision when redis an elasticsearch are ready.

@qiangxue

This comment has been minimized.

Show comment
Hide comment
@qiangxue

qiangxue Sep 14, 2013

Member

By having redis directly under the framework folder, it's like making it a self-contained extension with dependency on Yii. Other directories under db mainly contain drivers which are not the same as redis.

Member

qiangxue commented Sep 14, 2013

By having redis directly under the framework folder, it's like making it a self-contained extension with dependency on Yii. Other directories under db mainly contain drivers which are not the same as redis.

@cebe

This comment has been minimized.

Show comment
Hide comment
@cebe

cebe Sep 14, 2013

Member

Agree. Will make the move soon.

Member

cebe commented Sep 14, 2013

Agree. Will make the move soon.

@mcd-php

This comment has been minimized.

Show comment
Hide comment
@mcd-php

mcd-php Nov 21, 2013

Contributor

I vote for Apache Cassandra inclusion - it uses SQL-like language ( CQL ), has PDO driver, i made it work for my closed-source job project.

On the other side, my piece for it may be application-biased (or have some parts of application inside it), and ignorant of some framework guidelines. It's also ridden with backward-compatibility kludges, since i converted from Yii1 and from PHPCassa to PDO - two overhauls far before start ;).

So i it may be better if core team write their own, stumble upon all lack of dependency injection, and ask me questions if stuck. Or should i go over all my code and file issues on all stumbles i had to dodge ?

Anyway, i am willing to help with this.

Contributor

mcd-php commented Nov 21, 2013

I vote for Apache Cassandra inclusion - it uses SQL-like language ( CQL ), has PDO driver, i made it work for my closed-source job project.

On the other side, my piece for it may be application-biased (or have some parts of application inside it), and ignorant of some framework guidelines. It's also ridden with backward-compatibility kludges, since i converted from Yii1 and from PHPCassa to PDO - two overhauls far before start ;).

So i it may be better if core team write their own, stumble upon all lack of dependency injection, and ask me questions if stuck. Or should i go over all my code and file issues on all stumbles i had to dodge ?

Anyway, i am willing to help with this.

@qiangxue

This comment has been minimized.

Show comment
Hide comment
@qiangxue

qiangxue Nov 22, 2013

Member

It would be great if you could submit a PR for this, and we can work together to polish it (unless other @yiisoft/core-developers has experience with Cassandra and willing to work on this). Thanks.

Member

qiangxue commented Nov 22, 2013

It would be great if you could submit a PR for this, and we can work together to polish it (unless other @yiisoft/core-developers has experience with Cassandra and willing to work on this). Thanks.

@cebe

This comment has been minimized.

Show comment
Hide comment
@cebe

cebe Dec 17, 2013

Member

I think we can close this issue now. I am done with all the system I know and can support as I am going to work with them on a regular basis. Others should be provided as extensions or by other core developers. If someone wants to start working on an implementation of an AR feel free to contact me, I can help with that.

Member

cebe commented Dec 17, 2013

I think we can close this issue now. I am done with all the system I know and can support as I am going to work with them on a regular basis. Others should be provided as extensions or by other core developers. If someone wants to start working on an implementation of an AR feel free to contact me, I can help with that.

@cebe cebe closed this Dec 17, 2013

@rubenheymans

This comment has been minimized.

Show comment
Hide comment
@rubenheymans

rubenheymans Mar 9, 2015

@joltup any progress with Couchbase?

@joltup any progress with Couchbase?

@cebe

This comment has been minimized.

Show comment
Hide comment
@cebe

cebe Mar 9, 2015

Member

as far as I know nobody is working on that, if you want it you can start yourself and ask for help if you need.

Member

cebe commented Mar 9, 2015

as far as I know nobody is working on that, if you want it you can start yourself and ask for help if you need.

@TheHett TheHett referenced this issue in yiisoft/yii2-mongodb Aug 7, 2015

Closed

Why we not implement db transaction to mongodb? #29

This was referenced Apr 6, 2016

This was referenced Mar 14, 2018

samdark pushed a commit that referenced this issue Mar 30, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment