Online and streaming algorithms with Akka
For good overview of streaming algos go to this debasishg's gist.
In org.apache.spark.streamdm
there is code copied from Huawei Noah's Ark Lab streamDM project (the licencse is Apache 2.0 too). The code has been adapted to work with Akka streams instead of Spark streaming module by removing dependencies on the Spark stuff. Tested path is the HoeffdingTree
model usage, which works best with input sourced from Arff
files via SpecificationParser
/ ExampleParser
- see see other parts of akka-online
for usage examples.
If you'd like to know more about HoeffdingTree on-line classifier, please read the HDT docs or go to Massive Online Analysis website that contains more information on the context.
Sample Arff
files can be found in the MOA dataset repository.
For theoretical exposition to HoeffdingTree
usage go to the original paper.
If you are asking yourself question why Akka, not Spark? - please read the classic bigger data, same laptop to see what we can gain going lightweight. In previous century, 4.5k records machine learning data set could be considered sizeable and hence all the growth of clustering for data processing. But with today's hardware I believe we can do better.
In the /lib
directory of the project I have put suffixtree-1.0.0-SNAPSHOT.jar
which is a compiled artifact of the Ukkonen's on-line Suffix Tree implementation. The licensse for this project is also Apache 2.0.
In this project I also use Guava implementation of Bloom Filter
(see project dependencies) but there are other options.