New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/improvement suggestions for ckanext-realtime #56

Open
reederz opened this Issue May 20, 2014 · 5 comments

Comments

Projects
None yet
3 participants
@reederz

reederz commented May 20, 2014

I have been working on ckanext-realtime (https://github.com/alexandrainst/ckanext-realtime) for some time now and I have reached the initial goal for it. If you enable this extension on your CKAN instance, client applications can monitor changes to datastores in realtime. In the picture below you can see how the realtime datastores are enabled with this plugin.
screenshot - 05202014 - 01 54 40 pm

  1. In order to emit realtime events, each "realtime datastore" table has a PostgreSQL trigger which sends NOTIFY events to datastore_listener script. The insertion of the trigger to datastore tables happens automatically: first time someone is interested to monitor a particular datastore, that datastores gets the trigger.
  2. datastore_listener script sends datastore events (insert/update/delete) to ckan through Action API function realtime_broadcast_event.
  3. CKAN and WebSocket server(s) are integrated using Redis Pub/Sub messaging where CKAN is the publisher and WebSocket server(s) are the subscribers. This way, every event sent to realtime_broadcast_event API function ends up in WebSocket servers.
  4. Clients can subscribe to realtime events of certain datastores (pub/sub) using CkanRT.js library (included in ckanext-realtime repo).

So, right now it is possible for client apps to observe insert/delete/update events of certain datastores. You can find a demo app here http://gatesense.com/realtime/examples/ex2/ .

Other things that could be implemented:

  • dataset level realtime events (new resources in datasets, dataset metadata change, etc.)
  • direct integration into CKAN UI
  • access control
  • anything else?

The project was initiated by Alexandra Institute (http://www.alexandra.dk/uk/pages/default.aspx) and later adopted by Gatesense community (http://gatesense.com/) where Alexandra is one of the contributors. It is an effort to research realtime data publishing in the context of IoT and eventually gear CKAN towards being "IoT enabled" (though this may require additional extensions). That said, this extension has not been used on "real world applications" yet and it may not cover your specific use case. Therefore, we welcome community's collaborations and suggestions on how to improve it.

@stevenlivz

This comment has been minimized.

Show comment
Hide comment
@stevenlivz

stevenlivz May 21, 2014

This is excellent :-)

stevenlivz commented May 21, 2014

This is excellent :-)

@rufuspollock

This comment has been minimized.

Show comment
Hide comment
@rufuspollock

rufuspollock May 21, 2014

Contributor

+1 very cool!

Contributor

rufuspollock commented May 21, 2014

+1 very cool!

@rufuspollock

This comment has been minimized.

Show comment
Hide comment
@rufuspollock

rufuspollock May 30, 2014

Contributor

@reederz this is great. Do you think you could provide a short 1-2 paragraphs summarizing this work suitable for a broad audience (so not too technical) for inclusion in the next newsletter - you can put your info in the newsletter scratchpad google doc

Contributor

rufuspollock commented May 30, 2014

@reederz this is great. Do you think you could provide a short 1-2 paragraphs summarizing this work suitable for a broad audience (so not too technical) for inclusion in the next newsletter - you can put your info in the newsletter scratchpad google doc

@reederz

This comment has been minimized.

Show comment
Hide comment
@reederz

reederz May 30, 2014

@rgrp yes, I can do that. I reckon that I should do that today because you are releasing the newsletter soon, right?

reederz commented May 30, 2014

@rgrp yes, I can do that. I reckon that I should do that today because you are releasing the newsletter soon, right?

@rufuspollock

This comment has been minimized.

Show comment
Hide comment
@rufuspollock

rufuspollock May 30, 2014

Contributor

@reederz yes - i you could add something today or tomorrow that would be great.

Contributor

rufuspollock commented May 30, 2014

@reederz yes - i you could add something today or tomorrow that would be great.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment