GCS Hadoop Connector can't recover from Tablet Server failure.

[adamresson]

I was attempting to use the GCS Connector (https://github.com/GoogleCloudPlatform/bigdata-interop/tree/master/gcs) to back Accumulo on GCP.

All pretty straight forward (just pointed instance.volumes to my bucket gs://<bucketname>/accumulo

One of my Tablet Servers OOMed, and when I try to recover it, I end up getting the following error:

Failed to initiate log sort gs://<bucketname>/accumulo/wal/accumulo-gcs-w-1+9997/aa166493-637e-48e8-a9a6-3655dfb59a6c
	java.lang.IllegalStateException: Don't know how to recover a lease for com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem
		at org.apache.accumulo.server.master.recovery.HadoopLogCloser.close(HadoopLogCloser.java:70)
		at org.apache.accumulo.master.recovery.RecoveryManager$LogSortTask.run(RecoveryManager.java:96)
		at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
		at java.util.concurrent.FutureTask.run(FutureTask.java:266)
		at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
		at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
		at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
		at java.lang.Thread.run(Thread.java:748)

I traced that back to HadoopLogCloser, and since the GoogleHadoopFileSystem is a FileSystem, not a DistributedFileSystem, LocalFileSystem, RawLocalFileSystem, it bottoms out here:

accumulo/server/base/src/main/java/org/apache/accumulo/server/master/recovery/HadoopLogCloser.java

Line 71 in 5d216d4

"Don't know how to recover a lease for " + ns.getClass().getName());

I'm not sure what to do from here. I was also told that Azure should work with accumulo no problem, but it looks like the NativeAzureFileSystem similarly doesn't implement any of these interfaces, so I would presume it would hit the same issue.

[keith-turner]

I'm not sure what to do from here.

I think the following needs to be done.

• Determine how one process can close a file using GCSC such that any other processes writing to the file will fail. Then implement this as a org.apache.accumulo.server.master.recovery.LogCloser.
• Put this implementation on Accumulo classpath
• Set the accumulo property master.walog.closer.implementation to use the impl

I am really curious about the overall write ahead log behavior with GCSC. It would be interesting to intentionally kill tablet servers and see if the data recovers correctly.

[adamresson]

Looking at GCS, that WAL file doesn't even exist, so I'm not sure where it's storing that upon restar. Is it somewhere in zookeeper I can wipe out, at least to temporarily negate this issue?

[keith-turner]

Is it somewhere in zookeeper I can wipe out, at least to temporarily negate this issue?

Try looking in /accumulo/<uuid>/root_tablet/walogs in zookeeper.

[ctubbsii]

I'm not sure there's a general solution in Accumulo we can provide for this scenario. We can't really know how to react to unknown FileSystem implementations. If it's not one of the ones we support, the user must provide a LogCloser implementation for that specific FileSystem.

This is really more of a question, than an issue. I'm inclined to close it as answered. Does that seem reasonable?

[adamresson]

Yup, makes sense @ctubbsii . Closing.