Improved the I/O architecture documentation. (#13558)

Small cosmetic fixes.
Expanded the Sink.write() section with one more example.
Added SelectorManager diagram.

Signed-off-by: Simone Bordet <simone.bordet@gmail.com>

Author: sbordet

documentation/jetty/modules/code/examples/src/main/java/org/eclipse/jetty/docs/programming/ContentDocs.java

@@ -19,6 +19,7 @@
 import java.util.ArrayList;
 import java.util.List;
 import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ThreadLocalRandom;
 
 import org.eclipse.jetty.io.Content;
 import org.eclipse.jetty.util.Callback;
@@ -284,6 +285,7 @@ static class SinkMany
         public void manyWrites(Content.Sink sink, ByteBuffer content1, ByteBuffer content2)
         {
             // Initiate a first write.
+            // Callback.Completable is-a CompletableFuture.
             Callback.Completable resultOfWrites = Callback.Completable.with(callback1 -> sink.write(false, content1, callback1))
                 // Chain a second write only when the first is complete.
                 .compose(callback2 -> sink.write(true, content2, callback2));
@@ -301,8 +303,74 @@ public void manyWrites(Content.Sink sink, ByteBuffer content1, ByteBuffer conten
         // end::sinkMany[]
     }
 
-    // tag::copy[]
     @SuppressWarnings("InnerClassMayBeStatic")
+    // tag::sinkChunks[]
+    class LargeDownload extends IteratingCallback
+    {
+        public void download(Content.Sink sink, Callback callback)
+        {
+            // Create the IteratingCallback and start the iteration.
+            IteratingCallback writer = new LargeDownload(sink, callback, 1024 * 1024);
+            writer.iterate();
+        }
+
+        private static final int CHUNK_SIZE = 1024;
+        private final Content.Sink sink;
+        private final Callback callback;
+        private int length;
+
+        public LargeDownload(Content.Sink sink, Callback callback, int downloadLength)
+        {
+            this.sink = sink;
+            // The callback to notify when the download is completed.
+            this.callback = callback;
+            this.length = downloadLength;
+        }
+
+        @Override
+        protected Action process() throws Throwable
+        {
+            // Return when the whole download is completed.
+            if (length == 0)
+                return Action.SUCCEEDED;
+
+            // Prepare the chunk to write.
+            int size = Math.min(CHUNK_SIZE, length);
+            byte[] bytes = new byte[size];
+            ThreadLocalRandom.current().nextBytes(bytes);
+            ByteBuffer byteBuffer = ByteBuffer.wrap(bytes);
+            length -= size;
+            boolean last = length == 0;
+
+            // Start the non-blocking write, passing "this" as the callback.
+            sink.write(last, byteBuffer, this);
+            return Action.SCHEDULED;
+        }
+
+        @Override
+        protected void onCompleteSuccess()
+        {
+            // Download completed, notify the download callback.
+            callback.succeeded();
+        }
+
+        @Override
+        protected void onCompleteFailure(Throwable failure)
+        {
+            // Download failed, notify the download callback.
+            callback.failed(failure);
+        }
+
+        @Override
+        public InvocationType getInvocationType()
+        {
+            return InvocationType.NON_BLOCKING;
+        }
+    }
+    // end::sinkChunks[]
+
+    @SuppressWarnings("InnerClassMayBeStatic")
+    // tag::copy[]
     class Copy extends IteratingCallback
     {
         private final Content.Source source;
@@ -349,22 +417,22 @@ protected Action process() throws Throwable
         @Override
         protected void onSuccess()
         {
-            // After every successful write, release the chunk
-            // and reset to the next chunk
+            // After every successful write, release
+            // the chunk and reset to the next chunk.
             chunk = Content.Chunk.releaseAndNext(chunk);
         }
 
         @Override
         protected void onCompleteSuccess()
         {
-            // The copy is succeeded, succeed the callback.
+            // The copy is complete, succeed the copy callback.
             callback.succeeded();
         }
 
         @Override
         protected void onFailure(Throwable cause)
         {
-            // The copy is failed, fail the callback.
+            // The copy has failed, fail the copy callback.
             // This method is invoked before a write() has completed, so
             // the chunk is not released here, but in onCompleteFailure().
             callback.failed(cause);

documentation/jetty/modules/programming-guide/pages/arch/io.adoc

@@ -21,9 +21,41 @@ The Jetty libraries (both client and server) use Java NIO to handle I/O, so that
 The main class of The Jetty I/O library is link:{javadoc-url}/org/eclipse/jetty/io/SelectorManager.html[`SelectorManager`].
 
 `SelectorManager` manages internally a configurable number of link:{javadoc-url}/org/eclipse/jetty/io/ManagedSelector.html[`ManagedSelector`]s.
-Each `ManagedSelector` wraps an instance of `java.nio.channels.Selector` that in turn manages a number of `java.nio.channels.SocketChannel` instances.
+Each `ManagedSelector` wraps an instance of `java.nio.channels.Selector` that in turn manages a number of `java.nio.channels.SelectionKey` instances, that each wrap a `java.nio.channels.SelectableChannel` instance such as `java.nio.channels.SocketChannel`.
 
-NOTE: TODO: add image
+[plantuml]
+----
+object SM as "SelectorManager"
+object MS1 as "ManagedSelector" {
+  java.nio.channels.Selector
+}
+object MS2 as "ManagedSelector" {
+  java.nio.channels.Selector
+}
+object SC1A as "SelectionKey" {
+  SelectableChannel
+}
+object SC1B as "SelectionKey" {
+  SelectableChannel
+}
+object SC2A as "SelectionKey" {
+  SelectableChannel
+}
+object SC2B as "SelectionKey" {
+  SelectableChannel
+}
+object SC2C as "SelectionKey" {
+  SelectableChannel
+}
+
+SM o-- MS1
+SM o-- MS2
+MS1 o-- SC1A
+MS1 o-- SC1B
+MS2 o-- SC2A
+MS2 o-- SC2B
+MS2 o-- SC2C
+----
 
 `SocketChannel` instances are typically created by the Jetty implementation, on client-side when connecting to a server and on server-side when accepting connections from clients.
 In both cases the `SocketChannel` instance is passed to `SelectorManager` (which passes it to `ManagedSelector` and eventually to `java.nio.channels.Selector`) to be registered for use within Jetty.
@@ -90,7 +122,7 @@ In the Jetty I/O library, you can call `EndPoint.fillInterested(Callback)` to de
 At the Java NIO level, a `SocketChannel` or `DatagramChannel` is always writable, unless it becomes congested.
 In order to be notified when a channel uncongests and it is therefore writable again, the `SelectionKey.OP_WRITE` flag must be set.
 
-In the Jetty I/O library, you can call `EndPoint.write(Callback, ByteBuffer...)` to write the ``ByteBuffer``s and the `Callback` parameter is the object that is notified when the whole write is finished (i.e. _all_ ``ByteBuffer``s have been fully written, even if they are delayed by congestion/uncongestion).
+In the Jetty I/O library, you can call `EndPoint.write(Callback, ByteBuffer\...)` to write the ``ByteBuffer``s and the `Callback` parameter is the object that is notified when the whole write is finished (i.e. _all_ ``ByteBuffer``s have been fully written, even if they are delayed by congestion/uncongestion).
 
 The `EndPoint` APIs abstract out the Java NIO details by providing non-blocking APIs based on `Callback` objects for I/O operations.
 The `EndPoint` APIs are typically called by `Connection` implementations, see <<connection,this section>>.
@@ -176,7 +208,7 @@ Submitting the invocation of the callback to an `Executor` to be invoked in a di
 
 This side effect of asynchronous programming leading to `StackOverflowError` is so common that the Jetty libraries have a generic solution for it: a specialized `Callback` implementation named `org.eclipse.jetty.util.IteratingCallback` that turns recursion into iteration, therefore avoiding the `StackOverflowError`.
 
-`IteratingCallback` is a `Callback` implementation that should be passed to non-blocking APIs such as `EndPoint.write(Callback, ByteBuffer...)` when they are performed in a loop.
+`IteratingCallback` is a `Callback` implementation that should be passed to non-blocking APIs such as `EndPoint.write(Callback, ByteBuffer\...)` when they are performed in a loop.
 
 `IteratingCallback` works by starting the loop with `IteratingCallback.iterate()`.
 In turn, this calls `IteratingCallback.process()`, an abstract method that must be implemented with the code that should be executed for each loop.
@@ -197,11 +229,11 @@ include::code:example$src/main/java/org/eclipse/jetty/docs/programming/SelectorM
 ----
 
 When `onFillable()` is called, for example the first time that bytes are available from the network, the iteration is started.
-Starting the iteration calls `process()`, where a buffer is allocated and filled with bytes read from the network via `EndPoint.fill(ByteBuffer)`; the buffer is subsequently written back via `EndPoint.write(Callback, ByteBuffer...)` -- note that the callback passed to `EndPoint.write()` is `this`, i.e. the `IteratingCallback` itself; finally `Action.SCHEDULED` is returned, returning from the `process()` method.
+Starting the iteration calls `process()`, where a buffer is allocated and filled with bytes read from the network via `EndPoint.fill(ByteBuffer)`; the buffer is subsequently written back via `EndPoint.write(Callback, ByteBuffer\...)` -- note that the callback passed to `EndPoint.write()` is `this`, i.e. the `IteratingCallback` itself; finally `Action.SCHEDULED` is returned, returning from the `process()` method.
 
-At this point, the call to `EndPoint.write(Callback, ByteBuffer...)` may have completed synchronously; `IteratingCallback` would know that and call `process()` again; within `process()`, the buffer has already been allocated so it will be reused, saving further allocations; the buffer will be filled and possibly written again; `Action.SCHEDULED` is returned again, returning again from the `process()` method.
+At this point, the call to `EndPoint.write(Callback, ByteBuffer\...)` may have completed synchronously; `IteratingCallback` would know that and call `process()` again; within `process()`, the buffer has already been allocated so it will be reused, saving further allocations; the buffer will be filled and possibly written again; `Action.SCHEDULED` is returned again, returning again from the `process()` method.
 
-At this point, the call to `EndPoint.write(Callback, ByteBuffer...)` may have not completed synchronously, so `IteratingCallback` will not call `process()` again; the processing thread is free to return to the Jetty I/O system where it may be put back into the thread pool.
+At this point, the call to `EndPoint.write(Callback, ByteBuffer\...)` may have not completed synchronously, so `IteratingCallback` will not call `process()` again; the processing thread is free to return to the Jetty I/O system where it may be put back into the thread pool.
 If this was the only active network connection, the system would now be idle, with no threads blocked, waiting that the `write()` completes. This thread-less wait is one of the most important features that make non-blocking asynchronous servers more scalable: they use less resources.
 
 Eventually, the Jetty I/O system will notify that the `write()` completed; this notifies the `IteratingCallback` that can now resume the loop and call `process()` again.
@@ -360,8 +392,16 @@ include::code:example$src/main/java/org/eclipse/jetty/docs/programming/ContentDo
 
 When you need to perform an unknown number of writes, you may use an `IteratingCallback`, explained in <<echo,this section>>, to avoid ``StackOverFlowError``s.
 
-For example, to copy from a `Content.Source` to a `Content.Sink` you could use the convenience method `Content.copy(Content.Source, Content.Sink, Callback)`.
-For illustrative purposes, below you can find the implementation of `copy(Content.Source, Content.Sink, Callback)` that uses an `IteratingCallback`:
+For example, to download a large content in smaller chunks, you can do this:
+
+[,java,indent=0]
+----
+include::code:example$src/main/java/org/eclipse/jetty/docs/programming/ContentDocs.java[tags=sinkChunks]
+----
+
+If you want to copy from a `Content.Source` to a `Content.Sink` you could use the convenience method `Content.copy(Content.Source, Content.Sink, Callback)`.
+
+For illustrative purposes, below you can find a more elaborate example with an implementation of `copy(Content.Source, Content.Sink, Callback)` that uses `IteratingCallback`:
 
 [,java,indent=0]
 ----