Threads hang on put in closed/closing/opening

[yaitskov]

I have 3 threads stuck with the same stack trace.
Version is 0.2.2

I launched process inserting data into 1 table over 1 connection with 6 threads.
Right after that I truncated table with hbase shell.
I saw in log complains that table is not found.

exception #1548 from bigtable.googleapis.com for \x00\x04\x1A\x8F\x00\x00\x01H\x06\xD0}\x80A\x95
io.grpc.StatusRuntimeException: NOT_FOUND: Failed to read table: table1
        at io.grpc.Status.asRuntimeException(Status.java:430)
        at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:266)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$3.run(ClientCallImpl.java:320)
        at io.grpc.internal.SerializingExecutor$TaskRunner.run(SerializingExecutor.java:154)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
exception #1549 from bigtable.googleapis.com for \x00\x04\x1A\x8F\x00\x00\x01H\x06\xD0}\x80EU
io.grpc.StatusRuntimeException: UNKNOWN
        at io.grpc.Status.asRuntimeException(Status.java:430)
        at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:266)
        at io.grpc.ClientInterceptors$CheckedForwardingClientCall.start(ClientInterceptors.java:172)
        at io.grpc.stub.ClientCalls.startCall(ClientCalls.java:193)
        at io.grpc.stub.ClientCalls.asyncUnaryRequestCall(ClientCalls.java:173)
        at io.grpc.stub.ClientCalls.asyncUnaryRequestCall(ClientCalls.java:163)
        at io.grpc.stub.ClientCalls.asyncUnaryCall(ClientCalls.java:68)
        at com.google.cloud.bigtable.grpc.io.ClientCallService$1.listenableAsyncCall(ClientCallService.java:94)
        at com.google.cloud.bigtable.grpc.BigtableDataGrpcClient.mutateRowAsync(BigtableDataGrpcClient.java:150)
        at com.google.cloud.bigtable.grpc.async.AsyncExecutor$1.call(AsyncExecutor.java:58)
        at com.google.cloud.bigtable.grpc.async.AsyncExecutor$1.call(AsyncExecutor.java:55)
--
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalStateException: Channel is closed
        at com.google.cloud.bigtable.grpc.io.ReconnectingChannel$DelayingCall.start(ReconnectingChannel.java:88)
        at com.google.cloud.bigtable.grpc.io.ChannelPool$1.checkedStart(ChannelPool.java:97)
        at io.grpc.ClientInterceptors$CheckedForwardingClientCall.start(ClientInterceptors.java:164)
        ... 42 more

After one thread got RetriesExhaustedWithDetailsException it reopened the connection.

org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 27 actions: StatusRuntimeException: 27 times,
        at com.google.cloud.bigtable.hbase.BatchExecutor.batch(BatchExecutor.java:294)
        at com.google.cloud.bigtable.hbase.BatchExecutor.batch(BatchExecutor.java:318)
        at com.google.cloud.bigtable.hbase.BigtableTable.put(BigtableTable.java:280)

Rest threads access to closed/closing/opening connection. Maybe this was not good idea to do so,
but I think that threads shouldn't hang forever.

Stacktrace of hanging threads:

"pool-12-thread-5" prio=10 tid=0x00002aaab85c6800 nid=0x325c waiting on condition [0x0000000042eb7000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000079a5b03b8> (a com.google.bigtable.repackaged.com.google.common.util.concurrent.AbstractFuture$Sync)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
        at com.google.bigtable.repackaged.com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:285)
        at com.google.bigtable.repackaged.com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
        at com.google.cloud.bigtable.hbase.BatchExecutor.batch(BatchExecutor.java:280)
        at com.google.cloud.bigtable.hbase.BatchExecutor.batch(BatchExecutor.java:318)
        at com.google.cloud.bigtable.hbase.BigtableTable.put(BigtableTable.java:280)

[sduskis]

Unfortunately, truncating the table deletes and recreates the table, which would explain the NOT_FOUND behavior. We don't currently have a safe way to clean a table.

[sduskis]

The "java.lang.IllegalStateException: Channel is closed" indicates that you likely called connection.close() somewhere and then did further operations. We don't support that.

[sduskis]

You're right about the fact that we shouldn't be hanging in this situation. Is there any way you can provide a test case to reproduce this? It would help a lot to have that so we can make sure we can fix this case.

[yaitskov]

The algorithm is simple:

put into non-existing table
catch exception
close connection
use the closed connection put again

[sduskis]

Hm. We should just throw an illegal state exception. I'll add that. I'll also add a test.

[sduskis]

I merged the last week. I'll close this issue. If this is still a problem, please let me know.