Socket.io compatibility with cube.js

[rizvanhaider]

Is there any way to create the real-time charts with cubejs? compatibility of Socket.io or real-time data with cubejs will be a great feature because most analytics needs real-time data.

[nicodds]

I was evaluating the same thing. I would give my 2 cents on the topic suggesting the use of cossbar's Autobahn, since it is a well performing and multi-{platform|language} alternative to socket.io

[paveltiunov]

@rizvanhaider @nicodds Hey Rizwan and Domenico! Thanks for posting this! As it's ongoing conversation I'll put some more context info here.

In general most of databases supported by Cube.js are retroactive. It means databases usually don't support any push/subscription mechanism to listen for changes of underlying data. So to implement some sort of push/subscription behavior for clients cube.js backend should constantly poll databases for changes. Refresh key concept pretty well suited for that: https://cube.dev/docs/caching#in-memory-cache-refresh-keys.

So in order to implement it I think we need to decide on how should we define polling interval: should it be defined by client or server? Per cube or global?

@rizvanhaider @nicodds Would love to hear more about your socket push/subscriptions use cases to better understand what type of configuration it requires.

[nicodds]

Ciao @paveltiunov, thank you for the reply! My initial idea was to evaluate the possibility of integrating some sort of pub/sub component in cube.js backend. This integration may be used in two different ways:

1. notify subscribed clients of a change in a particular dataset, forcing reload;
   or
2. send directly the new data (or an aggregated form) to the subscribed clients.

[rizvanhaider]

@paveltiunov Thanks Pavel for your detailed response.
If we consider the use case of website analytics and we want to see the real time users.

Consider our own implementation of tracking users on our website and every new user make an increment in database (table => cube => dimension/measure), so If we want to see the real time figures via our cubejs api I think we need to implement it on specific cube and on specific measure/dimension. Also it should be defined by server so client should know it via subscription.

[paveltiunov]

@nicodds @rizvanhaider Cool! What DB do you use?

[nicodds]

Actually I use an custom hybrid solution built on a Bokeh (python) application with data read both from a sqlite file (pre-aggreations) and BigQuery. Updates are received by the application via Google Pub/Sub and sent via web sockets to the clients. Unfortunately, this solution is hard to maintain (Bokeh, although an interesting project, has a complex approach to build an application) and difficult to scale.

[paveltiunov]

@nicodds Got it! Do you plan to use external rollups https://cube.dev/docs/pre-aggregations#external-rollup? If so which DB?

[rizvanhaider]

@nicodds @rizvanhaider Cool! What DB do you use?

currently I am using MySQL.

[paveltiunov]

@rizvanhaider @nicodds Hey Guys! We released sockets support for preview. In order to use it just enable it on backend using env variable:

CUBEJS_WEB_SOCKETS=true

On client side we added WebSocketTransport which can be used as:

import cubejs from '@cubejs-client/core';
import WebSocketTransport from '@cubejs-client/ws-transport';

const cubejsApi = cubejs({
  transport: new WebSocketTransport({ authentication: CUBEJS_TOKEN, apiUrl: 'ws://localhost:4000/' })
})

We provided just simple reference implementation using low level ws api. There's SubscriptionServer class which can be used to build integrations with other socket libraries and frameworks like Socket.io or Serverless sockets: https://github.com/cube-js/cube.js/blob/master/packages/cubejs-api-gateway/SubscriptionServer.js.

We're open to various socket transport contributions and if you feel you need particular socket transport we can guide you on how to build it.

Could you please try it and let me know if it's helpful?

[nicodds]

That is super-cool! I'll give it a try ASAP!