Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement NoSQL Storage for ActiveRecord #11

Closed
cebe opened this issue Mar 24, 2013 · 71 comments
Closed

Implement NoSQL Storage for ActiveRecord #11

cebe opened this issue Mar 24, 2013 · 71 comments

Comments

@cebe
Copy link
Member

@cebe cebe commented Mar 24, 2013

Will implement:

redis (versions >= 2.6) (pull request: #905 merged)

elastic search (pull request: #1295 merged)

mongodb (pull request: #1438 merged)

Sphinx (Pull Request merged)

http://sphinxsearch.com/

Other systems that may get implemented (suggestions welcome):

Other resources:

@ghost ghost assigned cebe Mar 24, 2013
@cebe
Copy link
Member Author

@cebe cebe commented Mar 24, 2013

Btw the checkboxes in here are made like this: https://github.com/blog/1375-task-lists-in-gfm-issues-pulls-comments

@oherych
Copy link

@oherych oherych commented Mar 25, 2013

Most noSQL databases have connection via REST. Will be greate have one defined interface for all Rest client. This give us able have one selected (by developer) REST library for all needs. We just say with which library it should use

@Aoseala
Copy link
Contributor

@Aoseala Aoseala commented Mar 25, 2013

Personal think elasticsearch PHP client, Elastica is very good! this is his github repository URL: https://github.com/ruflin/Elastica

@ghost
Copy link

@ghost ghost commented Mar 25, 2013

You mentioned 2 of my yii extensions so I might share my thoughts/experiences on this issue here: The whole noSQL topic is far too complex to make a "one fits all" solution even possible (I've tried and failed :-) ). I wanted to make things easier with my ActiveResource extension but there are a lot of design flaws in there so it is a bad candidate for reviewing. My suggestion for Yii in this respect: Take a look at Guzzle which is a REST client framework. I takes care of the annoying parts (authentication, caching, asynchronous/parallel curl requests), is extensible via plugins and actively maintained. Yii could offer an interface on top of Guzzle so that there is a well defined API to develop against, maybe in combination with the repository pattern

@DaSourcerer
Copy link
Contributor

@DaSourcerer DaSourcerer commented Mar 25, 2013

As I see Guzzle being mentioned here: While Guzzle is indeed a formidable solution with facilities that make inclusion into Yii trivial, I'd like to advise caution. As mentioned before, it is not just a library but an entire framework. It would also introduce a hard dependency on libcurl (it is my impression that the original author simply wanted to spare himself from dealing with all the gory details of socket programming). Just saying 😏

@rawtaz
Copy link
Contributor

@rawtaz rawtaz commented Mar 25, 2013

This is almost getting a bit OT, but I have to say: +1 for the repository pattern. Been missing it in Yii a lot of times.

@PommeVerte
Copy link
Contributor

@PommeVerte PommeVerte commented Apr 15, 2013

I would love to see HyperDex in this list. I know this is a bit of a longshot but it's got good performance, supports ACID transactions and has strong consistency guarantees and fault tolerance to boot.

It's outperformed MongoDB, Cassandra, and I believe Redis in the YCSB (despite better consistency guarantees and fault tolerance).
It's only downside really is that it's relatively young compared to other solutions.

Here are some links:

http://hyperdex.org/doc/
http://hyperdex.org/performance/
http://hyperdex.org/performance/details/ (more details regarding benchmark)
https://github.com/tritondigital/php-hyperdex

Alternativly I could settle with MongoDB even though I'm uncomfortable with some aspects of the data storage (or non-storage in this case)

@klimov-paul
Copy link
Member

@klimov-paul klimov-paul commented Apr 23, 2013

Implement NoSQL Storage for ActiveRecord

I am not sure how exactly such implementation should look at the end.
I can not see an universal solution for this.
Different NOSQL storages use different approaches for the data storage and data structure.
There are several types:

  • key-value oriented (Redis)
  • document oriented (MongoDb)
  • graph oriented (Neo4j)

Key-value storages could match the original Yii model paradigm, which relies on architecture, when model has only (or at least mostly) scalar type attributes.

However document oriented storage does not fit the Yii model paradigm. In such storages (as MongoDb) the single storage entry is a huge nested JSON document, which can not be described by simple model with scalar attributes.

Graph oriented database (such as Neo4j) actually could fit the Yii model paradigm. However the search inside such storages is rather different from the common database. Here you are starting with some node and then go to the nodes related with it. Sometimes universal search criteria can not be composed for fetching particular node with the single query.
But the main problem with such storages as Neo4j is they do not support PHP directly and there are no PHP drivers for them. Such storages can be accessed only via REST API.

@cebe
Copy link
Member Author

@cebe cebe commented Apr 23, 2013

@Haensel commented in #73 (comment):
I don't know if it is still the case but if you are still planing to implement a mapper for graph databases (Neo4j, OrientDB etc.) I would suggest you to take a look at Tinkerpop Blueprints (https://github.com/tinkerpop/blueprints/wiki). From the wiki:

"Blueprints is analogous to the JDBC, but for graph databases. As such, it provides a common set of interfaces to allow developers to plug-and-play their graph database backend. Moreover, software written atop Blueprints works over all Blueprints-enabled graph databases."

Neo4j and OrientDb both support Blueprints and are the most widely used graph dbs, so it might be worth a look

@cebe
Copy link
Member Author

@cebe cebe commented May 4, 2013

In case you wonder where to find this: I'm on the redis branch: https://github.com/yiisoft/yii2/tree/redis

@PommeVerte
Copy link
Contributor

@PommeVerte PommeVerte commented May 12, 2013

Is there an interface or required functions for Connection class? I would like to implement support for Rexster (as per the tinkerpop comment earlier). I've already made a php client for RexPro (which part of is very similar to the Yii Connection.php). I'd like to use it as a helper as a first step even if it's redundant (because I might move on to making a php extension to support rexPro.
I'm going to start now just trying to stick as much as possible to the current file/functionality.

@cebe
Copy link
Member Author

@cebe cebe commented May 12, 2013

Is there an interface or required functions for Connection class?

Will be there but I am not finished with that yet. Will define the interface or base class, when we have more dbms implemented so I'd be very interested in seeing your class.

@PommeVerte
Copy link
Contributor

@PommeVerte PommeVerte commented May 14, 2013

ok!
I haven't gone very far yet. I've used your Connection class as a basis and have made one for Rexter. I've left out Transactions and the concept of Isolation (or non-isolation in this case which allows for queries to use bindings set by previous queries) because it will probably need a bit of discussion .
I'll get onto looking into activeRecord and ActiveQuery classes next. Once I get ideas as to what to do about representation of Vertices vs Edges and routes and so forth. I'll probably post my suggestions. (I also should have a look at Neo4Yii to see how Hansael tackled it)

@PommeVerte
Copy link
Contributor

@PommeVerte PommeVerte commented May 16, 2013

ok having a look around. How would you suggest modifying __get() and __set() for shema-less db structures in activeRecord? This is kind of tricky as it'll catch all.

@PommeVerte
Copy link
Contributor

@PommeVerte PommeVerte commented Jun 2, 2013

graph DB and gremlin discussion for graph implementation here : http://www.yiiframework.com/forum/index.php/topic/43863-graph-databases-gremlin-qquery-building/
I reckon I should have something to push online by the end of the week depending on how the discussion goes. a preview version if you will

@Sammaye
Copy link

@Sammaye Sammaye commented Jun 19, 2013

I don't mean to plug my own content here but: https://github.com/Sammaye/MongoYii can handle MongoDB in a schemaless manner and I have the intention of converting it to Yii2 as soon as a stable interface for its conversion is made.

Maybe a stripped down version of something like that could be used for core?

@cebe
Copy link
Member Author

@cebe cebe commented Jun 19, 2013

It's nothing bad about showing your own content when you think it is good. Looks really interesting, will definitifely take a look at it. Thanks for sharing!

@PommeVerte
Copy link
Contributor

@PommeVerte PommeVerte commented Aug 19, 2013

I've been very busy recently but finally shifted towards making Yii independent libraries for graph DBs. Mostly speaking a DB client for rexster and an equivalent of a QueryBuilder for gremlin. Now I'm concentrating on integrating these into Yii2. I was wondering where the correct place for external libraries to reside was? I've been sticking them in the helpers dir for now even though that didn't seem correct. Any help would be appreciated.
Thanks

@Sammaye
Copy link

@Sammaye Sammaye commented Aug 19, 2013

Normally, for better composer integration, many have taken to adding such things to a vendors folder. This pattern is, of course, also replicated in many languages including Java.

@PommeVerte
Copy link
Contributor

@PommeVerte PommeVerte commented Aug 19, 2013

Yeah, I couldn't seem to find one though. Doesn't matter, I'll place them there for now and move on. I'll move them around once I get a better idea.

@jabbon
Copy link

@jabbon jabbon commented Sep 6, 2013

Support for MongoDB +1 (ActiveRecord, Sessions)
Support for Redis +1 (Cache, Sessions)

@cebe
Copy link
Member Author

@cebe cebe commented Sep 7, 2013

Support for Redis +1 (Cache, Sessions)

@jabbon this is already implemented and available on master branch.

Support for MongoDB +1 (ActiveRecord, Sessions)

does it really make sense to store Sessions on MongoDB?

@Sammaye
Copy link

@Sammaye Sammaye commented Sep 7, 2013

Yes, MongoDB, if done right, can store the session completely in working set, i.e. RAM just like redis

@qiangxue
Copy link
Member

@qiangxue qiangxue commented Sep 14, 2013

Suggest moving /db/redis to /redis because redis is relatively independent of the other classes under db.

@cebe
Copy link
Member Author

@cebe cebe commented Sep 14, 2013

The current classes are but the AR does extend from the db\ActiveRecord
I think we should make a decision when redis an elasticsearch are ready.

@qiangxue
Copy link
Member

@qiangxue qiangxue commented Sep 14, 2013

By having redis directly under the framework folder, it's like making it a self-contained extension with dependency on Yii. Other directories under db mainly contain drivers which are not the same as redis.

This was referenced Mar 14, 2018
samdark pushed a commit that referenced this issue Mar 30, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
You can’t perform that action at this time.