Skip to content
This repository

Live Views: streaming queries #1227

Open
lvca opened this Issue December 24, 2012 · 8 comments

2 participants

Luca Garulli Mat Taylor
Luca Garulli
Owner

It would be cool having a way to provide streamable updates to the clients based on a sort of "Live Views". This idea comes from Ayman and it's very powerful!

Let's think at this simple example. A client is interested to this query:

select from order where customer.city.name = 'Rome' and amount > 100000

But it doesn't want to poll for new items, but rather be asynchronously notified when new entries match the query. Here we can apply the classic publish/subscribe pattern.

To handle it we could save 2 object in memory:
1) the target: Order
2) the filter: customer.city.name = 'Rome' and amount > 100000

In the new "OViewHook" class I could check for each view if the record's class is a sub-class of "target" (in this case Order). If this applies i could check the filter against the record. If it returns true it could be sent to the requester.

Other things to do are:

  • A good way to define the view. Probably a new asynchronous command like "STREAM select from order where customer.city.name = 'Rome' and amount > 100000"
  • On startup the hook must be installed and have to handle all the publish/subscribe mechanism
  • Provide a new REST way to access it. Something like /stream//. This has to support the asynchronous way
  • Update the JS driver to support it
Mat Taylor

Perhaps these 'live views' could be implenmted as an OrientDB plugin for ESPER. http://esper.codehaus.org.

Esper is a mature GPL java implementation of a complex event processing engine which can connect to several popular databases using a SQL like DSL for processing events with support for POJO event objects and sliding time windows. eg
select avg(price) from StockTickEvent.win:time(30 sec)

More details available at http://esper.codehaus.org/tutorials/tutorial/tutorial.html

Elasticsearch.org also has a similar feature they call 'percolation' (http://www.elasticsearch.org/blog/2011/02/08/percolator.html)

Mat Taylor

If esper is too heavyweight for this, (and too the GPL license too restrictive) at least the concept of 'windows' could be added to support simple event stream processing. The basic window views supported by ESPER are..

  • win:length,
  • win:length_batch,
  • win:time,
  • win:time_batch,
  • win:time_length_batch,
  • win:time_accum,
  • win:ext_timed, etc..

These properties could be perhaps be considered functional members of an OStream class, and the time related functions could take cron like timers as arguments.

More info here http://esper.codehaus.org/esper-4.7.0/doc/reference/en-US/html/epl_clauses.html

Luca Garulli
Owner

I've started to work to see how to implement this task as the best and I've a working prototype I can push on a separate branch. The trick was the usage of the Asynchronous Queries supported by OrientDB since release 0.8. Probably the hardest part now is to let Node.js and other drivers to support it.

Mat Taylor

Some more thoughts that might help on the RESTfull design side of this one.

The Atmosphere framework (https://github.com/Atmosphere/atmosphere) is a good OS java implmentation of websockets, comet, long polling http that might be usefull to extend OrientDB to support real time event streams over http.

NGINX also has a well designed streaming module that we use here
https://github.com/wandenberg/nginx-push-stream-module

To define a stream
POST /stream/{db} << {filter}

To publish to a stream
`POST /stream/{db}/{id} << {message}

To subscribe to a stream
GET /stream/{db}/{id}

Mat Taylor

What do you think about adding real time pub/sub support to the http api using long polling, comet, websockets etc..? The atmosphere framework could really help here.

Mat Taylor

Did this feature make it to the 1.4 branch? Are there any docs for this?

Mat Taylor

Could we get an update on why this feature was pushed back from 1.4 to 2.0 - is it just too complex?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.