[client] read submission for readonly handle should bypass OSE threads.

[sijie]

Descriptions of the changes in this PR:

Motivation

#1791 improves performance by bypassing OSE threads for read request submission.
however the change doesn't handle tailing ReadOnlyLedgerHandle

Changes

ReadOnlyLedgerHandle overrides isHandleWritable and return false always.

---

In order to uphold a high standard for quality for code contributions, Apache BookKeeper runs various precommit
checks for pull requests. A pull request can only be merged when it passes precommit checks. However running all
the precommit checks can take a long time, some trivial changes don't need to run all the precommit checks. You
can check following list to skip the tests that don't need to run for your pull request. Leave them unchecked if
you are not sure, committers will help you:

• [skip bookkeeper-server bookie tests]: skip testing org.apache.bookkeeper.bookie in bookkeeper-server module.
• [skip bookkeeper-server client tests]: skip testing org.apache.bookkeeper.client in bookkeeper-server module.
• [skip bookkeeper-server replication tests]: skip testing org.apache.bookkeeper.replication in bookkeeper-server module.
• [skip bookkeeper-server tls tests]: skip testing org.apache.bookkeeper.tls in bookkeeper-server module.
• [skip bookkeeper-server remaining tests]: skip testing all other tests in bookkeeper-server module.
• [skip integration tests]: skip docker based integration tests. if you make java code changes, you shouldn't skip integration tests.
• [skip build java8]: skip build on java8. ONLY skip this when ONLY changing files under documentation under site.
• [skip build java9]: skip build on java9. ONLY skip this when ONLY changing files under documentation under site.

---

---

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

If this PR is a BookKeeper Proposal (BP):

• Make sure the PR title is formatted like:
  <BP-#>: Description of bookkeeper proposal
e.g. BP-1: 64 bits ledger is support
• Attach the master issue link in the description of this PR.
• Attach the google doc link if the BP is written in Google Doc.

Otherwise:

• Make sure the PR title is formatted like:
  <Issue #>: Description of pull request
e.g. Issue 123: Description ...
• Make sure tests pass via mvn clean apache-rat:check install spotbugs:check.
• Replace <Issue #> in the title with the actual Issue number.

---

[sijie]

@nicmichael can you please review this PR to see if that matches your intention at #1791 ?

[eolivelli]

In client tests there are a lot of failure around fencing.
This is quite scary

[sijie]

retest this please

[eolivelli]

retest this please

[eolivelli]

there are still a lot of errors

[nicmichael]

@sijie Yes that was the intention.
I'm looking at the test failures now. Adding "|| isRecoveryRead" to isHandleWritable() in readEntriesInternalAsync seems to address some, but not all of the failures. I don't quite understand yet what's causing the failures.

[eolivelli]

maybe the errors on recovery are due to the fact that reads should strictly respect the order

[nicmichael]

During recovery I suppose they should...
In the absence of a recovery on a read-only LH, they don't need to, and getting rid of the ordering (and the context-switch) is a noticeable performance win. So the trick is to correctly distinguish these scenarios. Guess that's still not right.

[sijie]

yeah. I think we don't this change. we should just close it.

[nicmichael]

I think what we need is something like this:

        if (isHandleWritable() || metadata.isInRecovery()) { 
            // Ledger handle in read/write mode: submit to OSE for ordered execution.
            clientCtx.getMainWorkerPool().executeOrdered(ledgerId, op);
        } else {
            // Read-only ledger handle: bypass OSE and execute read directly in client thread.
            // This avoids a context-switch to OSE thread and thus reduces latency.
            op.run();
        }

It passes tests on my workstation. I'm just not quite sure whether this check needs any synchronization or can go without.

[sijie]

@nicmichael I think it is a bit risky to take recovery state into account. so I would rather rely on whether metadata is closed or not. that says, the check in current master is probably good enough for the cases we are targeting.

[nicmichael]

Sounds good to me. The most common case is probably addressed by the earlier change.

[eolivelli]

Okay, we don't need this.