Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Configuring snappy left minor compaction hung. #920

Closed
keith-turner opened this issue Jan 25, 2019 · 2 comments

Comments

@keith-turner
Copy link
Contributor

commented Jan 25, 2019

While runnning continuous ingest on EC2 using 2.0.0-alpha-2 I tried configuring snappy and this threw an exception. This seemed to have left the tablets in a bad state, because when I set compression back to gzip the tablets never tried to compact again. I had to kill the tservers. Ideally this problem would work itself out when config is updated.

Below are some of the exceptions seen on the tserver.

2019-01-24 17:03:25,319 [tablet.Tablet] ERROR: Failed to free tablet memory
java.lang.IllegalStateException
        at org.apache.accumulo.tserver.tablet.TabletMemory.finalizeMinC(TabletMemory.java:114)
        at org.apache.accumulo.tserver.tablet.Tablet.minorCompact(Tablet.java:937)
        at org.apache.accumulo.tserver.tablet.MinorCompactionTask.run(MinorCompactionTask.java:91)
        at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
        at org.apache.htrace.wrappers.TraceRunnable.run(TraceRunnable.java:57)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
        at java.lang.Thread.run(Thread.java:748)
2019-01-24 17:03:25,323 [tablet.MinorCompactionTask] ERROR: Unknown error during minor compaction for extent: 2<;7b6db6db6db6db81
java.lang.RuntimeException: java.lang.UnsatisfiedLinkError: org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z
        at org.apache.accumulo.tserver.tablet.Tablet.minorCompact(Tablet.java:933)
        at org.apache.accumulo.tserver.tablet.MinorCompactionTask.run(MinorCompactionTask.java:91)
        at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
        at org.apache.htrace.wrappers.TraceRunnable.run(TraceRunnable.java:57)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.UnsatisfiedLinkError: org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z
        at org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy(Native Method)
        at org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:63)
        at org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:136)
        at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:150)
        at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:168)
        at org.apache.accumulo.core.file.rfile.bcfile.Compression$Algorithm.getCompressor(Compression.java:648)
        at org.apache.accumulo.core.file.rfile.bcfile.BCFile$Writer$WBlockState.<init>(BCFile.java:150)
        at org.apache.accumulo.core.file.rfile.bcfile.BCFile$Writer.prepareMetaBlock(BCFile.java:389)
        at org.apache.accumulo.core.file.rfile.bcfile.BCFile$Writer.prepareMetaBlock(BCFile.java:409)
        at org.apache.accumulo.core.file.rfile.RFile$Writer.close(RFile.java:603)
        at org.apache.accumulo.tserver.tablet.Compactor.call(Compactor.java:272)
        at org.apache.accumulo.tserver.tablet.MinorCompactor.call(MinorCompactor.java:124)
        at org.apache.accumulo.tserver.tablet.Tablet.minorCompact(Tablet.java:918)
        ... 7 more
2019-01-24 17:03:25,324 [tserver.TabletServerResourceManager] ERROR: Thread "minor compactor 2" died java.lang.RuntimeException: java.lang.UnsatisfiedLinkError: org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z
java.lang.RuntimeException: java.lang.RuntimeException: java.lang.UnsatisfiedLinkError: org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z
        at org.apache.accumulo.tserver.tablet.MinorCompactionTask.run(MinorCompactionTask.java:107)
        at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
        at org.apache.htrace.wrappers.TraceRunnable.run(TraceRunnable.java:57)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: java.lang.UnsatisfiedLinkError: org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z
        at org.apache.accumulo.tserver.tablet.Tablet.minorCompact(Tablet.java:933)
        at org.apache.accumulo.tserver.tablet.MinorCompactionTask.run(MinorCompactionTask.java:91)
        ... 6 more
Caused by: java.lang.UnsatisfiedLinkError: org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z
        at org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy(Native Method)
        at org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:63)
        at org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:136)
        at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:150)
        at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:168)
        at org.apache.accumulo.core.file.rfile.bcfile.Compression$Algorithm.getCompressor(Compression.java:648)
        at org.apache.accumulo.core.file.rfile.bcfile.BCFile$Writer$WBlockState.<init>(BCFile.java:150)
        at org.apache.accumulo.core.file.rfile.bcfile.BCFile$Writer.prepareMetaBlock(BCFile.java:389)
        at org.apache.accumulo.core.file.rfile.bcfile.BCFile$Writer.prepareMetaBlock(BCFile.java:409)
        at org.apache.accumulo.core.file.rfile.RFile$Writer.close(RFile.java:603)
        at org.apache.accumulo.tserver.tablet.Compactor.call(Compactor.java:272)
        at org.apache.accumulo.tserver.tablet.MinorCompactor.call(MinorCompactor.java:124)
        at org.apache.accumulo.tserver.tablet.Tablet.minorCompact(Tablet.java:918)
        ... 7 more
@keith-turner

This comment has been minimized.

Copy link
Contributor Author

commented Jan 25, 2019

I think I tracked down the problem. The code retries for IOExceptions and RuntimeExceptions. However this was a UnsatisfiedLinkError which is neither. Need to see if this is a problem in 1.9.

keith-turner added a commit to keith-turner/accumulo that referenced this issue Jan 30, 2019

@ctubbsii

This comment has been minimized.

Copy link
Member

commented Jan 30, 2019

The UnsatisfiedLinkError should have been handled by the native-loading code in the Hadoop compression library we're using, and transformed into an IOException or another (non-Error) Exception. We might want to file a bug in upstream Hadoop regarding this... because the Hadoop compression API shouldn't be exposing the native code loading behavior from its internals via this Throwable.

@ctubbsii ctubbsii added this to Done in 2.0.0 Jun 14, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
2 participants
You can’t perform that action at this time.