Implementation Details

Communication Protocol

1. TCP

• Since TCP guarantees the delivery of the packets, there is no need to send whole current data and only changes(delta) can safely be sent.

2. UDP

• Currently accumulated result should be sent because if even one of the packets is lost, results eventually will be wrong. However, if full result is sent and it is lost on the way, next (more accurate) result will be send in the following timeout.

3. Optimization

• Even if results aren't changed, some indication for these tuples (0 for delta, current result for full reading) should be sent which is unnecessary.
  • Sending delta via TCP enables us to send only tuples whose values are changed and nonexist tuples in the delta can be assumed to be unchanged.
  • This trick can be applied with UDP because current result can be lost where group A is modified and then next results are successfully arrived but since there is no update for A and A isn't in the result, there will be inconsistencycompared to other groups in terms of online processing.

Communication Method

• In terms of private networks, communication method doesn't matter because it is assumed that client will talk to the cluster final node to subscribe to the results.

1. Pull

• Partial reading case makes updating results more complex.
  • Query calculates sum and average of some column at the same time. If the client reads only one of them, the other will possibly be different when it starts to read again.
  • To overcome this problem, there are simply 2 solutions:
    • Double buffering in the final node combined with done message that comes from client.
    • Don't allow partial reading. Whenever the client wants to read, send back whole results and then client can do whatever it wants on the whole data.

2. Push

• Final node sends the results as soon as they are generated. Therefore, intrinsically final node is just a collector for possibly many clients and clients does their own buffering according what they need.
  • It should be intrinsically multi-threaded because one thread should listen final node and modify the data structure while another thread is reading results according to the needs of the application.
  • There are 2 ways to buffering:
    • Locking on a common data structure such as circular list.
    • Double buffering where one is for writing updates and the other is for reading.