Push notifications on data changes (client-server data sync) #300
I'm submitting a
There is no way for now to make the client-side graph representation of the data aware of data changes. Clients have to query the server every time to be sure that data has not been updated since the last query. This opens to the possibility of the client using stale data.
hydrus should allows the clients to connect through WebSocket to push data changes for every object directly to the client.
Do you want to work on this issue?
This will probably be a task to be part of GSOC 2019
Please start collecting here ideas for implementation, tell the community how you would like to see implemented this feature.
The text was updated successfully, but these errors were encountered:
@Mec-iS shouldn't this work both ways. Rather than just making client aware of data changes on the server-side, client should be able to push changes to server as well (given it has been authorized). I don't know if
@vddesai1871 I think
Client does this by standard HTTP methods. (through operations available/provided in HydraDoc).
The best approach is something like a PubSub (with Redis as we have already it available) or better an inboxes system like the one used by actors in an Actor Model; the server and clients have each of them an inbox; messages are in a queue that is consumed by the actor. Types of messages can be for example: PUT this payload in resource with id, POST this payload in resource with id, GET resource with id.
After a bit of discussion with @vddesai1871 and inspiration from other comments here I would like to bring a solution into discussion that would use Websockets and an Inbox mechanism.
To implement the socket mechanism we would use FlaskSocketIO which is a Socket.io intregration for Flask applications and allows the client to be connected via a socket when it queries for the API Doc, also having connection inconsistency and etc.
Regarding the socket functionatily @vddesai1871 already developed an experiment and added some basic WebSocket support in hydrus and simulated a small client. You can run multiple clients and when making a request to hydrus it forwards the modification after successfully adding the resource in resource.py.
Below is the inbox that is kept by the server and by the clients.
This solution introduces modifications in both hydrus and the Hydra Python Agent. At hydrus it's necessary to add the table log mechanism as well as a web socket available to notify modifications at the table. Client side, it should hold an own internal table, connect to the web socket and also be able to handle the four different situations described below to maintain the data updated.
When starting a client, it will query the API basic structure and also copy the current server modification log table. After that, there are three situations that have to be addressed when dealing with new rows at the modification table: the client has internally an outdated resource that needs an update, the client never queried that resource and lastly the client was the one who made a transaction.
The client finds a JOB ID referring to an outdated resource
The client finds a new JOB ID, it queries it's Redis graph to check if it already has that specific resource,
The client finds a JOB ID to a nonexisting resource internally
The client finds a new JOB ID, it queries its Redis graph to check if it already has that specific resource, if not finding, meaning that the Client hasn't yet queried for that resource, the client can ignore the modification and simply add the row to its internal table since it's not relevant for the Client.
The Client finds a JOB ID made by itself on the table
When the Client finds a new job that isn't on its table, the first thing it has to do is to check if its internal resource has a Date signature after or similar to the one in the server table, if the internal client representation is more recent, basically the client adds that transaction to the table since the client has a more up to date resource that will be shown soon at the server table.
An important observation here is that, the Client should set the internal Date for an resource with the Date object sent as the response Header by the hydrus server(to make sure it uses an standard centralized date). Also, it should only set it's internal Date for a resource when receiving a successful response for the HTTP request sent to modify/create a resource on the server.
That's the overall concept. I've been trying to grasp more concepts from the Actor Model so the solution is more robust and can perhaps process some concurrent changes.
[Following discussing at https://github.com/HTTP-APIs/hydra-python-agent/pull/123]
The server has a limited sized modifications_table as the following:
In the end, hydrus has three core implementations:
This receives a PARAM with a Job ID and sends the table diff according to the last updated resource the Agent had.
A simulated server, compatible and working with current Agent PR, simulates this behavior and is available at: https://github.com/Guttz/simulated-hydrus-sync-socket.