Implement simple word generator-aggregator model on REEF

[DifferentSC]

As the first step of MIST project, we need to implement a simple word generator-aggregator stream processing application on REEF. This issue is important for two reasons below.

• We can measure the number of streaming queries which can be processed in the environment without ZooKeeper.
• We can practice development on REEF environment (especially Tang and Wake) and get used to some useful tools for building MIST like RemoteManager, NetworkService, etc. We can also think about how we can provide simple API to hide some complex things of REEF.

[TaeHunKim]

As a start, I'll implement a simple word counter with one task as mentioned in ISSUE #4 .

After that, I'll try to separate word generator and aggregator as two or three tasks.
(For this, I should study sharing data among tasks.)

[DifferentSC]

To separate the word generator and aggregator task, I think we'd better implement word generator and aggregator in different REEF jobs and run those jobs in separate machines.

We can use RemoteManager for the network communication. Because RemoteManager provides low-level socket-based API, we can make a connection easily if we have addresses of word generator and aggregator. It handles data using Wake EventHandler, so we can also easily implement receiving data process inside the EventHandler. The main aggregator task thread will sleep all the time, and EventHandler inside it will wake up and handle the data whenever it comes in. Word generator and aggregator should use same codec to be used for serialization / deserialization of the data.

Below is the overall structure of the model.

[bgchun]

@DifferentSC @TaeHunKim How about using NCS instead of RemoteManager?
@dafrista What do you think?

[DifferentSC]

I think NCS could be a better option, because each RM in Machine 2 should have different socket numbers and it could be troublesome to send data to all different sockets in Machine 1 manually.

[DifferentSC]

I made an example about how to use NCS in REEF Jobs. The code might be somewhat dirty, but I think it can be some help to resolving this issue.

https://github.com/DifferentSC/reef-ncs-example

[bgchun]

@DifferentSC We don't write dirty code. ;-)

[TaeHunKim]

Thanks to @DifferentSC, I success to implement word counter with one generator and one aggregator.
But I still have some questions and discussions about it. We may can talk about them f2f when you have time.
After that, I will refactor this code, make a pull request, and try to improve it with multiple aggregator.

[TaeHunKim]

I finished the basic experiment. I'll extend the experiment in other issued

[taegeonum]

@TaeHunKim Is it ok to close this issue? If yes, please close it.

[TaeHunKim]

@taegeonum I forgot to close it. Thank you. I'm going to close it.