Teku doesn't produce attestations failing with NoClassDefFoundError: Could not initialize class org.xerial.snappy.Snappy

[4u7]

Description

After upgrading Teku from 23.1.1 to 23.3.1 on Radxa 5b board, it stopped producing attestations. The log contains NoClassDefFoundError: Could not initialize class org.xerial.snappy.Snappy. I tried to downgrade the snappy library from 1.1.9.1 to 1.1.8.4 which was used in 23.1.1 and Teku started to work normally. I simply copied the library to the lib directory and updated classpath in the bin\teku. Looks like snappy 1.1.9.1 doesn't work on ARM properly.

Full stack trace:

2023-04-11 10:18:16.209+00:00 | beaconchain-async-3 | ERROR | GossipFailureLogger | Failed to publish attestation(s) for slot 6198689
java.util.concurrent.CompletionException: java.lang.NoClassDefFoundError: Could not initialize class org.xerial.snappy.Snappy
at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315) ~[?:?]
at java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:1194) ~[?:?]
at java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2309) ~[?:?]
at tech.pegasys.teku.infrastructure.async.SafeFuture.thenCompose(SafeFuture.java:548) ~[teku-infrastructure-async-23.3.1.jar:23.3.1]
at tech.pegasys.teku.networking.eth2.gossip.subnets.AttestationSubnetSubscriptions.gossip(AttestationSubnetSubscriptions.java:67) ~[teku-networking-eth2-23.3.1.jar:23.3.1]
at tech.pegasys.teku.networking.eth2.gossip.AttestationGossipManager.onNewAttestation(AttestationGossipManager.java:56) ~[teku-networking-eth2-23.3.1.jar:23.3.1]
at tech.pegasys.teku.networking.eth2.gossip.forks.versions.GossipForkSubscriptionsPhase0.publishAttestation(GossipForkSubscriptionsPhase0.java:229) ~[teku-networking-eth2-23.3.1.jar:23.3.1]
at tech.pegasys.teku.networking.eth2.gossip.forks.GossipForkManager.lambda$publishMessage$3(GossipForkManager.java:231) ~[teku-networking-eth2-23.3.1.jar:23.3.1]
at java.util.Optional.ifPresentOrElse(Optional.java:196) ~[?:?]
at tech.pegasys.teku.networking.eth2.gossip.forks.GossipForkManager.publishMessage(GossipForkManager.java:230) ~[teku-networking-eth2-23.3.1.jar:23.3.1]
at tech.pegasys.teku.networking.eth2.gossip.forks.GossipForkManager.publishAttestation(GossipForkManager.java:155) ~[teku-networking-eth2-23.3.1.jar:23.3.1]
at tech.pegasys.teku.infrastructure.subscribers.Subscribers.lambda$deliver$1(Subscribers.java:117) ~[teku-infrastructure-subscribers-23.3.1.jar:23.3.1]
at tech.pegasys.teku.infrastructure.subscribers.Subscribers.lambda$forEach$0(Subscribers.java:98) ~[teku-infrastructure-subscribers-23.3.1.jar:23.3.1]
at java.util.concurrent.ConcurrentHashMap$ValuesView.forEach(ConcurrentHashMap.java:4780) ~[?:?]
at tech.pegasys.teku.infrastructure.subscribers.Subscribers.forEach(Subscribers.java:95) ~[teku-infrastructure-subscribers-23.3.1.jar:23.3.1]
at tech.pegasys.teku.infrastructure.subscribers.Subscribers.deliver(Subscribers.java:117) ~[teku-infrastructure-subscribers-23.3.1.jar:23.3.1]
at tech.pegasys.teku.statetransition.attestation.AttestationManager.validateForGossipAndNotifySendSubscribers(AttestationManager.java:136) ~[teku-ethereum-statetransition-23.3.1.jar:23.3.1]
at tech.pegasys.teku.statetransition.attestation.AttestationManager.sendToSubscribersIfProducedLocally(AttestationManager.java:267) ~[teku-ethereum-statetransition-23.3.1.jar:23.3.1]
at tech.pegasys.teku.statetransition.attestation.AttestationManager.lambda$onAttestation$11(AttestationManager.java:240) ~[teku-ethereum-statetransition-23.3.1.jar:23.3.1]
at java.util.concurrent.CompletableFuture.uniApplyNow(CompletableFuture.java:684) ~[?:?]
at java.util.concurrent.CompletableFuture.uniApplyStage(CompletableFuture.java:662) ~[?:?]
at java.util.concurrent.CompletableFuture.thenApply(CompletableFuture.java:2168) ~[?:?]
at tech.pegasys.teku.infrastructure.async.SafeFuture.thenApply(SafeFuture.java:467) ~[teku-infrastructure-async-23.3.1.jar:23.3.1]
at tech.pegasys.teku.statetransition.attestation.AttestationManager.onAttestation(AttestationManager.java:221) ~[teku-ethereum-statetransition-23.3.1.jar:23.3.1]
at tech.pegasys.teku.statetransition.attestation.AttestationManager.lambda$processInternallyValidatedAttestation$6(AttestationManager.java:161) ~[teku-ethereum-statetransition-23.3.1.jar:23.3.1]
at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:718) ~[?:?]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?]
at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147) ~[?:?]
at tech.pegasys.teku.infrastructure.async.SafeFuture.lambda$propagateResult$3(SafeFuture.java:148) ~[teku-infrastructure-async-23.3.1.jar:23.3.1]
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?]
at java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:887) ~[?:?]
at java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2325) ~[?:?]
at tech.pegasys.teku.infrastructure.async.SafeFuture.whenComplete(SafeFuture.java:626) ~[teku-infrastructure-async-23.3.1.jar:23.3.1]
at tech.pegasys.teku.infrastructure.async.SafeFuture.whenComplete(SafeFuture.java:33) ~[teku-infrastructure-async-23.3.1.jar:23.3.1]
at tech.pegasys.teku.infrastructure.async.SafeFuture.propagateResult(SafeFuture.java:143) ~[teku-infrastructure-async-23.3.1.jar:23.3.1]
at tech.pegasys.teku.infrastructure.async.SafeFuture.lambda$exceptionallyCompose$34(SafeFuture.java:426) ~[teku-infrastructure-async-23.3.1.jar:23.3.1]
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?]
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?]
at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147) ~[?:?]
at tech.pegasys.teku.statetransition.validation.signatures.AggregatingSignatureVerificationService$SignatureTask.lambda$completeAsync$0(AggregatingSignatureVerificationService.java:259) ~[teku-ethereum-statetransition-23.3.1.jar:23.3.1]
at tech.pegasys.teku.infrastructure.async.SafeFuture.of(SafeFuture.java:82) ~[teku-infrastructure-async-23.3.1.jar:23.3.1]
at tech.pegasys.teku.infrastructure.async.AsyncRunner.lambda$runAsync$2(AsyncRunner.java:47) ~[teku-infrastructure-async-23.3.1.jar:23.3.1]
at tech.pegasys.teku.infrastructure.async.SafeFuture.of(SafeFuture.java:74) ~[teku-infrastructure-async-23.3.1.jar:23.3.1]
at tech.pegasys.teku.infrastructure.async.ScheduledExecutorAsyncRunner.lambda$createRunnableForAction$1(ScheduledExecutorAsyncRunner.java:124) ~[teku-infrastructure-async-23.3.1.jar:23.3.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.xerial.snappy.Snappy
at tech.pegasys.teku.networking.eth2.gossip.encoding.SnappyBlockCompressor.compress(SnappyBlockCompressor.java:46) ~[teku-networking-eth2-23.3.1.jar:23.3.1]
at tech.pegasys.teku.networking.eth2.gossip.encoding.SszSnappyEncoding.encode(SszSnappyEncoding.java:38) ~[teku-networking-eth2-23.3.1.jar:23.3.1]
at tech.pegasys.teku.networking.eth2.gossip.subnets.AttestationSubnetSubscriptions.lambda$gossip$0(AttestationSubnetSubscriptions.java:78) ~[teku-networking-eth2-23.3.1.jar:23.3.1]
at java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:1187) ~[?:?]
... 46 more

Versions (Add all that apply)

• Software version: teku/v23.3.1/linux-aarch_64/-privatebuild-openjdk64bitservervm-java-17
• Java version: OpenJDK 64-Bit Server VM (build 17.0.6+10-Ubuntu-0ubuntu120.04.1, mixed mode, sharing)
• OS Name & Version: [cat /etc/*release]

DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION="Ubuntu 20.04.6 LTS"
NAME="Ubuntu"
VERSION="20.04.6 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.6 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

[StefanBratanov]

Closing since there have not been other instances of this issue reported. We also raised an issue on the library side which is open: xerial/snappy-java#417 .

[StefanBratanov]

Hi @4u7 you can look in this issue xerial/snappy-java#417 reason for why you get this error

[StefanBratanov]

Hi @4u7 this issue seems to have been fixed by the latest version of snappy-java https://github.com/xerial/snappy-java/releases/tag/v1.1.10.0 We will include this new version in our next release.

[4u7]

Hi @StefanBratanov! Can confirm that upgrading snappy-java to 1.1.10.0 works fine. Thank you!