Browse files

Add rudimentary section for the new streaming API

  • Loading branch information...
Neverlord committed Aug 27, 2018
1 parent f76edc8 commit a4604f93c53fe57983e9247a01e9d366ad4cbc72
@@ -0,0 +1,100 @@
Streams in \lib describe data flow between actors. We are not aiming to provide
functionality similar to Apache projects like Spark, Flink or Storm. Likewise,
we have different goals than APIs such as RxJava, Reactive Streams, etc.
Streams complement asynchronous messages, request/response communication and
publish/subscribe in \lib. In a sense, actor streams in \lib are a building
block that users could leverage for building feature-complete stream
computation engines or reactive high-level Big Data APIs.
A stream establishes a logical channel between two or more actors for
exchanging a potentially unbound sequence of values. This channel uses demand
signaling to guarantee that senders cannot overload receivers.
\singlefig{stream}{Streaming Concept}{fig-stream}
Streams are directed and data flows only \emph{downstream}, i.e., from sender
(source) to receiver (sink). Establishing a stream requires a handshake in
order to initialize required state and signal initial demand.
\singlefig{stream-roles}{Streaming Roles}{fig-stream-roles}
\lib distinguishes between three roles in a stream: (1) a \emph{source} creates
streams and generates data, (2) a \emph{stage} transforms or filters data, and
(3) a \emph{sink} terminates streams by consuming data.
We usually draw streams as pipelines for simplicity. However, sources can have
any number of outputs (downstream actors). Likewise, sinks can have any number
of inputs (upstream actors) and stages can multiplex N inputs to M outputs.
Hence, streaming topologies in \lib support arbitrary complexity with forks and
\subsection{Stream Managers}
Streaming-related messages are handled separately. Under the hood, actors
delegate to \emph{stream managers} that in turn allow customization of their
behavior with \emph{drivers} and \emph{downstream managers}.
\singlefig{stream-manager}{Internals of Stream Managers}{fig-stream-manager}
Users usually can skip implementing driver classes and instead use the
lambda-based interface showcased in the following sections. Drivers implement
the streaming logic by taking inputs from upstream actors and pushing data to
the downstream manager. A source has no input buffer. Hence, drivers only
provide a \emph{generator} function that downstream managers call according to
A downstream manager is responsible for dispatching data to downstream actors.
The default implementation broadcasts data, i.e., all downstream actors receive
the same data. The downstream manager can also perform any sort multi- or
anycast. For example, a load-balancer would use an anycast policy to dispatch
data to the next available worker.
\subsection{Defining Sources}
The simplest way to defining a source is to use the \lstinline^make_source^
function and pass it three arguments: \emph{initializer} for the state,
\emph{generator} for producing values, and \emph{predicate} for signaling the
end of the stream.
\subsection{Defining Stages}
The function \lstinline^make_stage^ also takes three lambdas but additionally
the received input stream handshake as first argument. Instead of a predicate,
\lstinline^make_stage^ only takes a finalizer, since the stage does not produce
data on its own and a stream terminates if no more sources exist.
\subsection{Defining Sinks}
The function \lstinline^make_sink^ is similar to \lstinline^make_stage^, except
that is does not produce outputs.
\subsection{Initiating Streams}
In our example, we always have a source \lstinline^int_source^ and a sink
\lstinline^int_sink^ with an optional stage \lstinline^int_selector^. Sending
\lstinline^open_atom^ to the source initiates the stream and the source will
respond with a stream handshake.
Using the actor composition in \lib (\lstinline^snk * src^ reads \emph{sink
after source}) allows us to redirect the stream handshake we send in
\lstinline^caf_main^ to the sink (or to the stage and then from the stage to
the sink).
@@ -132,6 +132,7 @@ \part{Core Library}
\part{I/O Library}
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.

0 comments on commit a4604f9

Please sign in to comment.