Issue #1791: Read Submission should bypass OSE Threads

### Motivation

Profiling of our Bookkeeper Client code for read requests shows that client threads spend half of their time in dispatching requests to OrderedExecutors (just the dispatch itself, not the execution inside OSE): 54% of their CPU time is spent in OrderedExecutor.executeOrdered() (called by LedgerHandle.readEntriesInternalAsync()). The high time spend in request submission to OSE is largely caused by Linux scheduling cost, that is the cost of dispatching the OSE thread to CPU: 42% of total time (3/4th of executeOrdered() time), threads spend in Unsafe.unpark(), which is essentially Linux scheduling/dispatching of another thread.

### Changes

This change executes read submissions (PendingReadOp) on read-only ledger handles directly inside the client thread instead of submitting them to Ordered Executors.

Tests with a prototype have shown significant improvements in both overall CPU consumption as well as read latency. The additional work client threads have to do (the dispatch of the read requests to netty) is roughly the same as the (saved) dispatch cost to OSE, so the change turns out to be neutral for CPU consumption of client threads. In some experiments, the savings even exceed the additional work, and client threads consume less cpu even though they "do more". It also frees up lots of resources in OSE threads. Since it eliminates one context-switch in read submission and also avoids serialization of reads to the same ledger (or ledgers hashing to the same OSE), it also reduces read latency. For a mixed read-write workload (14,000 reads/sec on read-only ledgers, 4,000 writes/sec on another set of ledgers), this change has reduced CPU consumption of OSE threads by 25%, kept CPU consumption of client (and Netty) threads the same, and yielded a 6% improvement of read latency (as measured by BK Client).

Master Issue: #1791: Read Submission should bypass OSE Threads


Reviewers: Enrico Olivelli <eolivelli@gmail.com>, Andrey Yegorov <None>, Sijie Guo <sijie@apache.org>, Matteo Merli <mmerli@apache.org>

This closes #1792 from nicmichael/DirectRead, closes #1791

bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerHandle.java

@@ -840,7 +840,14 @@ CompletableFuture<LedgerEntries> readEntriesInternalAsync(long firstEntry,
                 ws.recycle();
             }
 
-            clientCtx.getMainWorkerPool().executeOrdered(ledgerId, op);
+            if (isHandleWritable()) {
+                // Ledger handle in read/write mode: submit to OSE for ordered execution.
+                clientCtx.getMainWorkerPool().executeOrdered(ledgerId, op);
+            } else {
+                // Read-only ledger handle: bypass OSE and execute read directly in client thread.
+                // This avoids a context-switch to OSE thread and thus reduces latency.
+                op.run();
+            }
         } else {
             op.future().completeExceptionally(BKException.create(ClientClosedException));
         }