+doc akka#17337: java doc samples

akka-docs-dev/rst/java/stream-composition.rst

@@ -0,0 +1,310 @@
+.. _composition-java:
+
+Modularity, Composition and Hierarchy
+=====================================
+
+Akka Streams provide a uniform model of stream processing graphs, which allows flexible composition of reusable
+components. In this chapter we show how these look like from the conceptual and API perspective, demonstrating
+the modularity aspects of the library.
+
+Basics of composition and modularity
+------------------------------------
+
+Every processing stage used in Akka Streams can be imagined as a "box" with input and output ports where elements to
+be processed arrive and leave the stage. In this view, a :class:`Source` is nothing else than a "box" with a single
+output port, or, a :class:`BidiFlow` is a "box" with exactly two input and two output ports. In the figure below
+we illustrate the most common used stages viewed as "boxes".
+
+|
+
+.. image:: ../images/compose_shapes.png
+   :align: center
+
+|
+
+The *linear* stages are :class:`Source`, :class:`Sink`
+and :class:`Flow`, as these can be used to compose strict chains of processing stages.
+Fan-in and Fan-out stages have usually multiple input or multiple output ports, therefore they allow to build
+more complex graph layouts, not just chains. :class:`BidiFlow` stages are usually useful in IO related tasks, where
+there are input and output channels to be handled. Due to the specific shape of :class:`BidiFlow` it is easy to
+stack them on top of each other to build a layered protocol for example. The ``TLS`` support in Akka is for example
+implemented as a :class:`BidiFlow`.
+
+These reusable components already allow the creation of complex processing networks. What we
+have seen so far does not implement modularity though. It is desirable for example to package up a larger graph entity into
+a reusable component which hides its internals only exposing the ports that are meant to the users of the module
+to interact with. One good example is the ``Http`` server component, which is encoded internally as a
+:class:`BidiFlow` which interfaces with the client TCP connection using an input-output port pair accepting and sending
+:class:`ByteString` s, while its upper ports emit and receive :class:`HttpRequest` and :class:`HttpResponse` instances.
+
+The following figure demonstrates various composite stages, that contain various other type of stages internally, but
+hiding them behind a *shape* that looks like a :class:`Source`, :class:`Flow`, etc.
+
+|
+
+.. image:: ../images/compose_composites.png
+   :align: center
+
+|
+
+One interesting example above is a :class:`Flow` which is composed of a disconnected :class:`Sink` and :class:`Source`.
+This can be achieved by using the ``wrap()`` constructor method on :class:`Flow` which takes the two parts as
+parameters.
+
+The example :class:`BidiFlow` demonstrates that internally a module can be of arbitrary complexity, and the exposed
+ports can be wired in flexible ways. The only constraint is that all the ports of enclosed modules must be either
+connected to each other, or exposed as interface ports, and the number of such ports needs to match the requirement
+of the shape, for example a :class:`Source` allows only one exposed output port, the rest of the internal ports must
+be properly connected.
+
+These mechanics allow arbitrary nesting of modules. For example the following figure demonstrates a :class:`RunnableGraph`
+that is built from a composite :class:`Source` and a composite :class:`Sink` (which in turn contains a composite
+:class:`Flow`).
+
+|
+
+.. image:: ../images/compose_nested_flow.png
+   :align: center
+
+|
+
+The above diagram contains one more shape that we have not seen yet, which is called :class:`RunnableGraph`. It turns
+out, that if we wire all exposed ports together, so that no more open ports remain, we get a module that is *closed*.
+This is what the :class:`RunnableGraph` class represents. This is the shape that a :class:`Materializer` can take
+and turn into a network of running entities that perform the task described. In fact, a :class:`RunnableGraph` is a
+module itself, and (maybe somewhat surprisingly) it can be used as part of larger graphs. It is rarely useful to embed
+a closed graph shape in a larger graph (since it becomes an isolated island as there are no open port for communication
+with the rest of the graph), but this demonstrates the uniform underlying model.
+
+If we try to build a code snippet that corresponds to the above diagram, our first try might look like this:
+
+.. includecode:: ../../../akka-samples/akka-docs-java-lambda/src/test/java/docs/stream/CompositionDocTest.java#non-nested-flow
+
+It is clear however that there is no nesting present in our first attempt, since the library cannot figure out
+where we intended to put composite module boundaries, it is our responsibility to do that. If we are using the
+DSL provided by the :class:`Flow`, :class:`Source`, :class:`Sink` classes then nesting can be achieved by calling one of the
+methods ``withAttributes()`` or ``named()`` (where the latter is just a shorthand for adding a name attribute).
+
+The following code demonstrates how to achieve the desired nesting:
+
+.. includecode:: ../../../akka-samples/akka-docs-java-lambda/src/test/java/docs/stream/CompositionDocTest.java#nested-flow
+
+Once we have hidden the internals of our components, they act like any other built-in component of similar shape. If
+we hide some of the internals of our composites, the result looks just like if any other predefine component has been
+used:
+
+|
+
+.. image:: ../images/compose_nested_flow_opaque.png
+   :align: center
+
+|
+
+If we look at usage of built-in components, and our custom components, there is no difference in usage as the code
+snippet below demonstrates.
+
+.. includecode:: ../../../akka-samples/akka-docs-java-lambda/src/test/java/docs/stream/CompositionDocTest.java#reuse
+
+Composing complex systems
+-------------------------
+
+In the previous section we explored the possibility of composition, and hierarchy, but we stayed away from non-linear,
+generalized graph components. There is nothing in Akka Streams though that enforces that stream processing layouts
+can only be linear. The DSL for :class:`Source` and friends is optimized for creating such linear chains, as they are
+the most common in practice. There is a more advanced DSL for building complex graphs, that can be used if more
+flexibility is needed. We will see that the difference between the two DSLs is only on the surface: the concepts they
+operate on are uniform across all DSLs and fit together nicely.
+
+As a first example, let's look at a more complex layout:
+
+|
+
+.. image:: ../images/compose_graph.png
+   :align: center
+
+|
+
+The diagram shows a :class:`RunnableGraph` (remember, if there are no unwired ports, the graph is closed, and therefore
+can be materialized) that encapsulates a non-trivial stream processing network. It contains fan-in, fan-out stages,
+directed and non-directed cycles. The ``closed()`` method of the :class:`FlowGraph` factory object allows the creation of a
+general closed graph. For example the network on the diagram can be realized like this:
+
+.. includecode:: ../../../akka-samples/akka-docs-java-lambda/src/test/java/docs/stream/CompositionDocTest.java#complex-graph
+
+In the code above we used the implicit port numbering feature to make the graph more readable and similar to the diagram.
+It is possible to refer to the ports, so another version might look like this:
+
+.. includecode:: ../../../akka-samples/akka-docs-java-lambda/src/test/java/docs/stream/CompositionDocTest.java#complex-graph-alt
+
+|
+
+Similar to the case in the first section, so far we have not considered modularity. We created a complex graph, but
+the layout is flat, not modularized. We will modify our example, and create a reusable component with the graph DSL.
+The way to do it is to use the ``partial()`` method on :class:`FlowGraph` factory. If we remove the sources and sinks
+from the previous example, what remains is a partial graph:
+
+|
+
+.. image:: ../images/compose_graph_partial.png
+   :align: center
+
+|
+
+We can recreate a similar graph in code, using the DSL in a similar way than before:
+
+.. includecode:: ../../../akka-samples/akka-docs-java-lambda/src/test/java/docs/stream/CompositionDocTest.java#partial-graph
+
+The only new addition is the return value of the builder block, which is a :class:`Shape`. All graphs (including
+:class:`Source`, :class:`BidiFlow`, etc) have a shape, which encodes the *typed* ports of the module. In our example
+there is exactly one input and output port left, so we can declare it to have a :class:`FlowShape` by returning an
+instance of it. While it is possible to create new :class:`Shape` types, it is usually recommended to use one of the
+matching built-in ones.
+
+The resulting graph is already a properly wrapped module, so there is no need to call `named()` to encapsulate the graph, but
+it is a good practice to give names to modules to help debugging.
+
+|
+
+.. image:: ../images/compose_graph_shape.png
+   :align: center
+
+|
+
+Since our partial graph has the right shape, it can be already used in the simpler, linear DSL:
+
+.. includecode:: ../../../akka-samples/akka-docs-java-lambda/src/test/java/docs/stream/CompositionDocTest.java#partial-use
+
+It is not possible to use it as a :class:`Flow` yet, though (i.e. we cannot call ``.filter()`` on it), but :class:`Flow`
+has a ``wrap()`` method that just adds the DSL to a :class:`FlowShape`. There are similar methods on :class:`Source`,
+:class:`Sink` and :class:`BidiShape`, so it is easy to get back to the simpler DSL if a graph has the right shape.
+For convenience, it is also possible to skip the partial graph creation, and use one of the convenience creator methods.
+To demonstrate this, we will create the following graph:
+
+|
+
+.. image:: ../images/compose_graph_flow.png
+   :align: center
+
+|
+
+The code version of the above closed graph might look like this:
+
+.. includecode:: ../../../akka-samples/akka-docs-java-lambda/src/test/java/docs/stream/CompositionDocTest.java#partial-flow-dsl
+
+.. note::
+  All graph builder sections check if the resulting graph has all ports connected except the exposed ones and will
+  throw an exception if this is violated.
+
+We are still in debt of demonstrating that :class:`RunnableGraph` is a component just like any other, which can
+be embedded in graphs. In the following snippet we embed one closed graph in another:
+
+.. includecode:: ../../../akka-samples/akka-docs-java-lambda/src/test/java/docs/stream/CompositionDocTest.java#embed-closed
+
+The type of the imported module indicates that the imported module has a :class:`ClosedShape`, and so we are not
+able to wire it to anything else inside the enclosing closed graph. Nevertheless, this "island" is embedded properly,
+and will be materialized just like any other module that is part of the graph.
+
+As we have demonstrated, the two DSLs are fully interoperable, as they encode a similar nested structure of "boxes with
+ports", it is only the DSLs that differ to be as much powerful as possible on the given abstraction level. It is possible
+to embed complex graphs in the fluid DSL, and it is just as easy to import and embed a :class:`Flow`, etc, in a larger,
+complex structure.
+
+We have also seen, that every module has a :class:`Shape` (for example a :class:`Sink` has a :class:`SinkShape`)
+independently which DSL was used to create it. This uniform representation enables the rich composability of various
+stream processing entities in a convenient way.
+
+Materialized values
+-------------------
+
+After realizing that :class:`RunnableGraph` is nothing more than a module with no unused ports (it is an island), it becomes clear that
+after materialization the only way to communicate with the running stream processing logic is via some side-channel.
+This side channel is represented as a *materialized value*. The situation is similar to :class:`Actor` s, where the
+:class:`Props` instance describes the actor logic, but it is the call to ``actorOf()`` that creates an actually running
+actor, and returns an :class:`ActorRef` that can be used to communicate with the running actor itself. Since the
+:class:`Props` can be reused, each call will return a different reference.
+
+When it comes to streams, each materialization creates a new running network corresponding to the blueprint that was
+encoded in the provided :class:`RunnableGraph`. To be able to interact with the running network, each materialization
+needs to return a different object that provides the necessary interaction capabilities. In other words, the
+:class:`RunnableGraph` can be seen as a factory, which creates:
+
+  * a network of running processing entities, inaccessible from the outside
+  * a materialized value, optionally providing a controlled interaction capability with the network
+
+Unlike actors though, each of the processing stages might provide a materialized value, so when we compose multiple
+stages or modules, we need to combine the materialized value as well (there are default rules which make this easier,
+for example `to()` and `via()` takes care of the most common case of taking the materialized value to the left.
+See :ref:`flow-combine-mat-scala` for details). We demonstrate how this works by a code example and a diagram which
+graphically demonstrates what is happening.
+
+The propagation of the individual materialized values from the enclosed modules towards the top will look like this:
+
+|
+
+.. image:: ../images/compose_mat.png
+  :align: center
+
+|
+
+To implement the above, first, we create a composite :class:`Source`, where the enclosed :class:`Source` have a
+materialized type of :class:`Promise<BoxedUnit>`. By using the combiner function ``Keep.left()``, the resulting materialized
+type is of the nested module (indicated by the color *red* on the diagram):
+
+.. includecode:: ../../../akka-samples/akka-docs-java-lambda/src/test/java/docs/stream/CompositionDocTest.java#mat-combine-1
+
+Next, we create a composite :class:`Flow` from two smaller components. Here, the second enclosed :class:`Flow` has a
+materialized type of :class:`Future<OutgoingConnection>`, and we propagate this to the parent by using ``Keep.right()``
+as the combiner function (indicated by the color *yellow* on the diagram):
+
+.. includecode:: ../../../akka-samples/akka-docs-java-lambda/src/test/java/docs/stream/CompositionDocTest.java#mat-combine-2
+
+As a third step, we create a composite :class:`Sink`, using our ``nestedFlow`` as a building block. In this snippet, both
+the enclosed :class:`Flow` and the folding :class:`Sink` has a materialized value that is interesting for us, so
+we use ``Keep.both()`` to get a :class:`Pair` of them as the materialized type of ``nestedSink`` (indicated by the color
+*blue* on the diagram)
+
+.. includecode:: ../../../akka-samples/akka-docs-java-lambda/src/test/java/docs/stream/CompositionDocTest.java#mat-combine-3
+
+As the last example, we wire together ``nestedSource`` and ``nestedSink`` and we use a custom combiner function to
+create a yet another materialized type of the resulting :class:`RunnableGraph`. This combiner function just ignores
+the :class:`Future<Sink>` part, and wraps the other two values in a custom case class :class:`MyClass`
+(indicated by color *purple* on the diagram):
+
+.. includecode:: ../../../akka-samples/akka-docs-java-lambda/src/test/java/docs/stream/CompositionDocTest.java#mat-combine-4a
+
+.. includecode:: ../../../akka-samples/akka-docs-java-lambda/src/test/java/docs/stream/CompositionDocTest.java#mat-combine-4b
+
+.. note::
+  The nested structure in the above example is not necessary for combining the materialized values, it just
+  demonstrates how the two features work together. See :ref:`flow-combine-mat-java` for further examples
+  of combining materialized values without nesting and hierarchy involved.
+
+Attributes
+----------
+
+We have seen that we can use ``named()`` to introduce a nesting level in the fluid DSL (and also explicit nesting by using
+``partial()`` from :class:`FlowGraph`). Apart from having the effect of adding a nesting level, ``named()`` is actually
+a shorthand for calling `withAttributes(Attributes.name("someName"))`. Attributes provide a way to fine-tune certain
+aspects of the materialized running entity. For example buffer sizes can be controlled via attributes (see
+:ref:`stream-buffers-scala`). When it comes to hierarchic composition, attributes are inherited by nested modules,
+unless they override them with a custom value.
+
+The code below, a modification of an earlier example sets the ``inputBuffer`` attribute on certain modules, but not
+on others:
+
+.. includecode:: ../../../akka-samples/akka-docs-java-lambda/src/test/java/docs/stream/CompositionDocTest.java#attributes-inheritance
+
+The effect is, that each module inherits the ``inputBuffer`` attribute from its enclosing parent, unless it has
+the same attribute explicitly set. ``nestedSource`` gets the default attributes from the materializer itself. ``nestedSink``
+on the other hand has this attribute set, so it will be used by all nested modules. ``nestedFlow`` will inherit from ``nestedSource``
+except the ``map`` stage which has again an explicitly provided attribute overriding the inherited one.
+
+|
+
+.. image:: ../images/compose_attributes.png
+   :align: center
+
+|
+
+This diagram illustrates the inheritance process for the example code (representing the materializer default attributes
+as the color *red*, the attributes set on ``nestedSink`` as *blue* and the attributes set on ``nestedFlow`` as *green*).

akka-docs-dev/rst/java/stream-flows-and-basics.rst

@@ -212,6 +212,8 @@ which will be running on the thread pools they have been configured to run on -
    Reusing *instances* of linear computation stages (Source, Sink, Flow) inside FlowGraphs is legal,
    yet will materialize that stage multiple times.
 
+.. _flow-combine-mat-java:
+
 Combining materialized values
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 

akka-docs-dev/rst/java/stream-index.rst

@@ -11,6 +11,7 @@ Streams
    ../stream-design
    stream-flows-and-basics
    stream-graphs
+   stream-composition
    stream-rate
    stream-customize
    stream-integrations