No description, website, or topics provided.
Java Scala
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
dstream-api
dstream-dev-template
dstream-nifi
dstream-spark
dstream-sql
dstream-tez
gradle/wrapper
.gitignore
LICENSE
README.md
build.gradle
gradle.properties
gradlew
gradlew.bat
settings.gradle

README.md

DStream - Distributable Streams

==========

IMPORTANT: At the moment this is a research project with the primary goal of investigating the feasability of the approach.

The primary focus of the DStream API is to provide a Stream-based unified programming model to build ETL-style distributable data processes to be executed in compatible execution environments. While agnostic to any specific type of execution environment, the API aims to provide an extensible integration/delegation model to support variaty of execution environments.

The key distinction between Java 8 Stream and DStream is the notion of distributable data, which implies that the actual data may or may not be distributed, making DStream somewhat of a universal strategy to build ETL-style processes regardless of the location and/or the type of data as well as the execution environment it will be processed in.

The following code snippet shows an example of a quintessential WordCount:

Future<Stream<Stream<Entry<String, Integer>>>> resultFuture = DStream.ofType(String.class, "wc")
    .flatMap(line -> Stream.of(line.split("\\s+")))
    .reduceValues(word -> word, word -> 1, Integer::sum)
  .executeAs("WordCount");

// each stream within a stream represents a partition essentially giving you access 
// to each result partition
Stream<Stream<Entry<String, Integer>>> result = resultFuture.get();
result.forEach(resultPartitionStream -> {
   resultPartitionStream.forEach(System.out::println);
});
result.close();

Producing output similar to this:

We=4
cannot=2
created=3
our=1
problems=2
same=1
solve=1
the=4
. . .

======

For features overviewo please follow Core Features Overview

To get started please follow Getting Started

=======