Implement NoSQL Storage for ActiveRecord

[cebe]

Will implement:

redis (versions >= 2.6) (pull request: #905 merged)

• http://redis.io/topics/protocol
• https://github.com/nrk/predis/blob/v0.7/lib/Predis/Network/StreamConnection.php#L70
• https://github.com/Shumkov/Rediska
• https://github.com/ptrofimov/tinyredisclient/blob/master/src/TinyRedisClient.php
• phpnode: https://github.com/phpnode/YiiRedis/blob/master/ARedisRecord.php
• https://github.com/jdp/redisent/blob/master/src/redisent/Redis.php
• Cluster: https://github.com/colinmollenhour/credis
• Redis Connection Class
• Redis Active Record Class
• Redis Active Query Class
• Redis Extended Active Query Class (where, order, ...)
• Redis Relation Concept
• Redis Relation Implementation
• Redis Transaction Class (will not be implemented as retrieving values from redis is not possible in a transaction)
• Redis Cache (ready on master)
• Redis Session (via yii\web\CacheSession and yii\caching\RedisCache)

elastic search (pull request: #1295 merged)

• elastic search: https://github.com/phpnode/YiiBlocks/tree/master/elasticSearch
• Connection Class
• Active Record Class
• Active Query Class
• Relation Concept
• Relation Implementation
• Transaction Class(?)

mongodb (pull request: #1438 merged)

• http://docs.mongodb.org/manual/reference/sql-comparison/
• http://docs.mongodb.org/ecosystem/drivers/php-libraries/
• http://snmaynard.com/2012/10/17/things-i-wish-i-knew-about-mongodb-a-year-ago/
• http://www.yiiframework.com/extension/mongoyii/
• http://php.net/manual/en/class.mongoclient.php
• https://github.com/Sammaye/MongoYii
• https://github.com/canni/YiiMongoDbSuite
• Connection Class
• Active Record Class
• Active Query Class
• Relation Concept
• Relation Implementation
• Transaction Class(?)

Sphinx (Pull Request merged)

http://sphinxsearch.com/

Other systems that may get implemented (suggestions welcome):

• solr: https://github.com/phpnode/YiiSolr
• neo4j: https://github.com/Haensel/Neo4Yii
• couchdb: http://couchdb.apache.org/
• orientdb
  • http://www.orientdb.org/
  • https://github.com/AntonTerekhov/OrientDB-PHP
  • http://odino.org/graph-in-php-through-orientdb/
  • https://github.com/nuvolabase/orientdb

Other resources:

• https://github.com/Haensel/ActiveResource

[cebe]

Btw the checkboxes in here are made like this: https://github.com/blog/1375-task-lists-in-gfm-issues-pulls-comments

[oherych]

Most noSQL databases have connection via REST. Will be greate have one defined interface for all Rest client. This give us able have one selected (by developer) REST library for all needs. We just say with which library it should use

[Aoseala]

Personal think elasticsearch PHP client, Elastica is very good! this is his github repository URL: https://github.com/ruflin/Elastica

[ghost]

You mentioned 2 of my yii extensions so I might share my thoughts/experiences on this issue here: The whole noSQL topic is far too complex to make a "one fits all" solution even possible (I've tried and failed :-) ). I wanted to make things easier with my ActiveResource extension but there are a lot of design flaws in there so it is a bad candidate for reviewing. My suggestion for Yii in this respect: Take a look at Guzzle which is a REST client framework. I takes care of the annoying parts (authentication, caching, asynchronous/parallel curl requests), is extensible via plugins and actively maintained. Yii could offer an interface on top of Guzzle so that there is a well defined API to develop against, maybe in combination with the repository pattern

[DaSourcerer]

As I see Guzzle being mentioned here: While Guzzle is indeed a formidable solution with facilities that make inclusion into Yii trivial, I'd like to advise caution. As mentioned before, it is not just a library but an entire framework. It would also introduce a hard dependency on libcurl (it is my impression that the original author simply wanted to spare himself from dealing with all the gory details of socket programming). Just saying 😏

[rawtaz]

This is almost getting a bit OT, but I have to say: +1 for the repository pattern. Been missing it in Yii a lot of times.

[dmill-bz]

I would love to see HyperDex in this list. I know this is a bit of a longshot but it's got good performance, supports ACID transactions and has strong consistency guarantees and fault tolerance to boot.

It's outperformed MongoDB, Cassandra, and I believe Redis in the YCSB (despite better consistency guarantees and fault tolerance).
It's only downside really is that it's relatively young compared to other solutions.

Here are some links:

http://hyperdex.org/doc/
http://hyperdex.org/performance/
http://hyperdex.org/performance/details/ (more details regarding benchmark)
https://github.com/tritondigital/php-hyperdex

Alternativly I could settle with MongoDB even though I'm uncomfortable with some aspects of the data storage (or non-storage in this case)

[klimov-paul]

Implement NoSQL Storage for ActiveRecord

I am not sure how exactly such implementation should look at the end.
I can not see an universal solution for this.
Different NOSQL storages use different approaches for the data storage and data structure.
There are several types:

• key-value oriented (Redis)
• document oriented (MongoDb)
• graph oriented (Neo4j)

Key-value storages could match the original Yii model paradigm, which relies on architecture, when model has only (or at least mostly) scalar type attributes.

However document oriented storage does not fit the Yii model paradigm. In such storages (as MongoDb) the single storage entry is a huge nested JSON document, which can not be described by simple model with scalar attributes.

Graph oriented database (such as Neo4j) actually could fit the Yii model paradigm. However the search inside such storages is rather different from the common database. Here you are starting with some node and then go to the nodes related with it. Sometimes universal search criteria can not be composed for fetching particular node with the single query.
But the main problem with such storages as Neo4j is they do not support PHP directly and there are no PHP drivers for them. Such storages can be accessed only via REST API.

[cebe]

@Haensel commented in #73 (comment):
I don't know if it is still the case but if you are still planing to implement a mapper for graph databases (Neo4j, OrientDB etc.) I would suggest you to take a look at Tinkerpop Blueprints (https://github.com/tinkerpop/blueprints/wiki). From the wiki:

"Blueprints is analogous to the JDBC, but for graph databases. As such, it provides a common set of interfaces to allow developers to plug-and-play their graph database backend. Moreover, software written atop Blueprints works over all Blueprints-enabled graph databases."

Neo4j and OrientDb both support Blueprints and are the most widely used graph dbs, so it might be worth a look

[cebe]

In case you wonder where to find this: I'm on the redis branch: https://github.com/yiisoft/yii2/tree/redis

[dmill-bz]

Is there an interface or required functions for Connection class? I would like to implement support for Rexster (as per the tinkerpop comment earlier). I've already made a php client for RexPro (which part of is very similar to the Yii Connection.php). I'd like to use it as a helper as a first step even if it's redundant (because I might move on to making a php extension to support rexPro.
I'm going to start now just trying to stick as much as possible to the current file/functionality.

[cebe]

Is there an interface or required functions for Connection class?

Will be there but I am not finished with that yet. Will define the interface or base class, when we have more dbms implemented so I'd be very interested in seeing your class.

[dmill-bz]

ok!
I haven't gone very far yet. I've used your Connection class as a basis and have made one for Rexter. I've left out Transactions and the concept of Isolation (or non-isolation in this case which allows for queries to use bindings set by previous queries) because it will probably need a bit of discussion .
I'll get onto looking into activeRecord and ActiveQuery classes next. Once I get ideas as to what to do about representation of Vertices vs Edges and routes and so forth. I'll probably post my suggestions. (I also should have a look at Neo4Yii to see how Hansael tackled it)

[dmill-bz]

ok having a look around. How would you suggest modifying __get() and __set() for shema-less db structures in activeRecord? This is kind of tricky as it'll catch all.

[dmill-bz]

graph DB and gremlin discussion for graph implementation here : http://www.yiiframework.com/forum/index.php/topic/43863-graph-databases-gremlin-qquery-building/
I reckon I should have something to push online by the end of the week depending on how the discussion goes. a preview version if you will

[cebe]

Sadly haven't had the time to go there, but there is some really interesting stuff at berlinbuzzwords conference this year http://berlinbuzzwords.de/ .

http://berlinbuzzwords.de/program/session-schedule/all

• http://berlinbuzzwords.de/sessions/document-relations-elasticsearch
• http://de.slideshare.net/martijnvg/document-relationsbbuz2013
• http://berlinbuzzwords.de/sessions/couchdb-implementation

[Sammaye]

I don't mean to plug my own content here but: https://github.com/Sammaye/MongoYii can handle MongoDB in a schemaless manner and I have the intention of converting it to Yii2 as soon as a stable interface for its conversion is made.

Maybe a stripped down version of something like that could be used for core?

[cebe]

It's nothing bad about showing your own content when you think it is good. Looks really interesting, will definitifely take a look at it. Thanks for sharing!

[dmill-bz]

I've been very busy recently but finally shifted towards making Yii independent libraries for graph DBs. Mostly speaking a DB client for rexster and an equivalent of a QueryBuilder for gremlin. Now I'm concentrating on integrating these into Yii2. I was wondering where the correct place for external libraries to reside was? I've been sticking them in the helpers dir for now even though that didn't seem correct. Any help would be appreciated.
Thanks

[Sammaye]

Normally, for better composer integration, many have taken to adding such things to a vendors folder. This pattern is, of course, also replicated in many languages including Java.

[dmill-bz]

Yeah, I couldn't seem to find one though. Doesn't matter, I'll place them there for now and move on. I'll move them around once I get a better idea.

[jabbon]

Support for MongoDB +1 (ActiveRecord, Sessions)
Support for Redis +1 (Cache, Sessions)

[cebe]

Support for Redis +1 (Cache, Sessions)

@jabbon this is already implemented and available on master branch.

Support for MongoDB +1 (ActiveRecord, Sessions)

does it really make sense to store Sessions on MongoDB?

[Sammaye]

Yes, MongoDB, if done right, can store the session completely in working set, i.e. RAM just like redis

[qiangxue]

Suggest moving /db/redis to /redis because redis is relatively independent of the other classes under db.

[cebe]

The current classes are but the AR does extend from the db\ActiveRecord
I think we should make a decision when redis an elasticsearch are ready.

[qiangxue]

By having redis directly under the framework folder, it's like making it a self-contained extension with dependency on Yii. Other directories under db mainly contain drivers which are not the same as redis.