Stack Random ideas

Main Design Parts

1. How to combine results at one node? {Comment: why you need results on a single node has to be explained first}

• Final node can be a part of the topology:
  • Storm automatically creates the node so knowing its place beforehand seems impossible. However, there are some updates in Storm(0.8.0) that enables manual topology configuration. In some clusters like in EPFL cluster, worker nodes can not be accessible from outside, so results must be collected at a node that is accessible.
  • Since node is a part of topology, it intrinsically knows the place of its parents to get updates.
• Final node can be outside of the topology:
  • Node should learn its parents to get updates. It seems difficult because of dynamic nature of Storm. {Comment: the main issue is about deciding on frequency of sent data as close to Squall final nodes as possible} Moreover, for further goal, nested query support, batch execution may be combined with online processing later. {Comment: this sentence is unclear.}
• Putting node into topology seems more reasonable.

2. How is the client going to get results from combiner node?

• Push
  • Client will be lighter since there is no need to setup timers.
  • Client can adjust the rate. If the rate is higher than what the client wants, the client can discard arriving tuples. However, if the rate is slower than expected, it can not be increased. Moreover, here, there is an implicit assumption, Squall does its best and sends tuples as soon as they arrive. Since client may discard tuples (rate is high), there is inefficient usage of network resources.
  • Public
    • From communication point of view, there is no problem that is seen.
  • Private
    • From communication point of view, there are problems because client can be behind of a NAT, etc.
• Pull
  • Client can adjust the rate on its own that results to utilize network more efficiently.
  • Private or Public:
    • Since final node (result combiner) of Squall is public, there is no problem from communication point of view.

3. How is the update rate (timeout) selected?

• Currently, 200ms is used which is somewhat a magic number. {Comment: this is used only within an example, it's not a general rule. You'll come up with better heuristics.}
• Increase:
  • Updates will be shown later so online fashion is lost.
• Decrease:
  • Updates will be tried to be sent faster. Since nodes in the execution pipeline tries to communicate to other nodes, network will be bottleneck which increases execution time (throughput decreases).

4. What is data structure of getting updates?

• Main caching of result tuples can be done in the final node or in the parents of the final node.
  • If caching is done in the parent, synchronization is needed since parent will write new arriving result tuples and final node(combiner) will read some parts of the results. This is favorable if each parent generates one group by and final node enables clients to view some portions of the group bys.
  • In the other way, final node will get tuples as much as its parents send. However, since multiple parents send and only one combiner makes the cache, it will be the bottleneck. {Comment: you can have multiple final nodes and achieve different tradeoffs.} Parents can asynchronously send tuples and don't bother whether combiner gets the tuples because according to timeout (chosen at 3), parents will send more accurate results in very near future. {Comment: try to rewrite the last sentence, unclear}