Stack Random ideas

Main Design Parts

0. why we need results to be collected in a single node? - Reason behind why an API is needed

• According to the query plan, final results can be produced by multiple nodes. However, client application which is outside of the Storm topology is generally composed of only one node. Therefore, client must handle the overflow of the result stream in some way, dropping, buffering, etc.
• A client must know the places of the nodes that are producing the result tuples to collect them. However, since it is outside of the topology, this requires query based configuration which is cumbersome.
• A client doesn't need to be bothered with collecting the results, it just wants to get updates via talking a proxy in the topology. Therefore, result stream must be collected and saved at the proxy for the clients to retrieve it by one to one communication.
• Selecting this proxy node configuration will be cluster based, not the query based.

1. How to combine results at one node?

• Final node can be a part of the topology:
  • Storm automatically creates the node so knowing its place beforehand seems impossible. However, there are some updates in Storm(0.8.0) that enables manual topology configuration. In some clusters like in EPFL cluster, worker nodes can not be accessible from outside, so results must be collected at a node that is accessible.
  • Since node is a part of topology, it intrinsically knows the place of its parents to get updates.
• Final node can be outside of the topology:
  • Node should learn its parents to get updates. It seems difficult because of dynamic nature of Storm. The point is more than only getting the places of the parents, final node must be aware of the speed of its parents to get updates as fast as they can produce which is crucial not to lose online execution property.
• Putting node into topology seems more reasonable.

2. How is the client going to get results from combiner node?

• Push
  • Client will be lighter since there is no need to setup timers.
  • Client can adjust the rate. If the rate is higher than what the client wants, the client can discard arriving tuples. However, if the rate is slower than expected, it can not be increased. Moreover, here, there is an implicit assumption, Squall does its best and sends tuples as soon as they arrive. Since client may discard tuples (rate is high), there is inefficient usage of network resources.
  • Public
    • From communication point of view, there is no problem that is seen.
  • Private
    • From communication point of view, there are problems because client can be behind of a NAT, etc.
• Pull
  • Client can adjust the rate on its own that results to utilize network more efficiently.
  • Private or Public:
    • Since final node (result combiner) of Squall is public, there is no problem from communication point of view.

3. How is the update rate (timeout) selected?

• Update rate shows a trade-off between execution speed (due to network latency) and online execution (seeing results of executed tuples instantly). Currently, 200 ms is used since it is a good balance of the factors but according to the method of collecting the results it will analyzed to come up with better one.
• Increase:
  • Updates will be shown later so online fashion is lost.
• Decrease:
  • Updates will be tried to be sent faster. Since nodes in the execution pipeline tries to communicate to other nodes, network will be bottleneck which increases execution time (throughput decreases).

4. What is data structure of getting updates?

• Main caching of result tuples can be done in the final node or in the parents of the final node.
  • If caching is done in the parent, synchronization is needed since parent will write new arriving result tuples and final node(combiner) will read some parts of the results. This is favorable if each parent generates one group by and final node enables clients to view some portions of the group bys. Another advantage is that since worker nodes still keep the tuples, nested query execution may be supported later.
  • In the other way, final node will get tuples as much as its parents send. However, since multiple parents send and only one combiner makes the cache, it will be the bottleneck. Parents will probably send much faster than the combiner can handle, if combiner can get the result tuple, it is fine and it will be put into data structure to be retrieved by the clients (this is case that the combiner can keep up with its parents). Otherwise, when the update timer of the parent times out, it will send again more accurate tuple since more input tuples are executed but there is no guarantee. The combiner node can successfully get this tuple so two update timeouts are seen by the combiner or if it is repeated more than once (means that misses again), the combiner will see multiple timeout slowdown. However, some round robin execution must be guaranteed because the combiner may not get tuples from some parents. Multiple final nodes in a tree structure can be used to get rid of this bottleneck and root of the tree will be the window to the outside. This increases complexity in result collection phase but the speed of the results will be more comparable to the speed of Squall result stream.