This can happen when we attempt to connect to a RegionServer, and we buffer some RPCs to send once connected, but we never successfully connect, because of a connection timeout. Bug reported by Viral Bajaria.
Add a new pair of public APIs to HBaseRpc to allow specifying whether an RPC should fail as soon as it encounters a problem. This is useful for applications with low-latency requirements where retries are unnecessary anyway. Once HBase has the ability to track a timeout per RPC this will be less useful, but in the interim it's a very useful and small addition.
Add an integration test to ensure this didn't break in the process.
Signed-off-by: Benoit Sigoure <email@example.com>
The code was wrong but worked by chance because the second code was set to 0 (NOT_ENCODED), and HBase actually uses the second code to know what type the object really is. So this is more of a "cosmetic issue on the wire", but better fix it anyway. Problem reported by Viral Bajaria.
If a .META. or -ROOT- lookup for a probe request leads to a NonRecoverableException (eg. when the region server serving either is down, leading to an underlying RegionServerStoppedException), the probe's callback (RetryRPC) is not executed. Now, if an upstream request comes in and sets the meta cache to a stale entry, all further requests will go to the wrong the regionserver and NSRE. The cache entry for the region is never deleted as that is the responsibility of the probe, which is now "lost"! The bug manifests itself at the client in various ways, like the client hanging up on regionserver restarts or continuos PleaseThrottleExceptions even after the Region is back online. This patch fixes the issue by triggering the callback when there is a NonRecoverableException for a request. Signed-off-by: Benoit Sigoure <firstname.lastname@example.org>
Because these atomic increments can't be buffered, they go out immediately, and client.flush() may not flush anything (because the RPCs are have already been sent). This is working as intended but it makes the test flaky. So make sure we really only verify the final count after the two increments have completed.
The problem was twofold: - flush() was ignoring the fact that some RPCs could be waiting for a -ROOT- lookup. If flush() is called early enough, all RPCs are in this situation. - flush() was ignoring the fact that some RegionClient in the processing of connecting and handshaking with the RegionServer could have some writes in its pending_rpcs list. This closes #2.
Use an internal interface (not part of the public API) to mark RPC classes that mutate data in HBase.
This closes #41, whereby a too large number of increments that pile up for a given counter could cause a Deferred chain to blow up. We now guarantee that this will not happen by keeping track of how many times each counter gets incremented, and pro-actively flushing those that reach the limit on the length of the Deferred chain. This has a nice side effect that the *really* busy counters will get flushed more often. This tracking is done without any extra locking or synchronization and only costs an extra 32 bits of memory per coalesced counter. This enhances increment coalescing to support negative increment values as well as values greater than Short.MAX_VALUE (65535). There is still a limit on how big the values can be, but the limit is much higher now (2^48 - 1) and values greater than that are still gracefully handled. The code is also more robust now in face of overflows / underflows, as the previous implementation had one potential race condition, which although it would have been incredibly rare, was still possible anyway. We now explicitly check for overflows / underflows before we CAS the value, so that when an overflow / underflow condition is detected we abort the update gracefully and let the caller know so they can flush the counter and create a new one to start coalescing again. This change also adds extra integration tests for various corner cases.