HazelcastSerializationException with Hazelcast 4.2 and bucket4j 6.2.0

[Jacuo]

I have upgraded my Hazelcast and bucket 4j. And now I have error
com.hazelcast.nio.serialization.HazelcastSerializationException: java.lang.ClassNotFoundException: io.github.bucket4j.grid.CommandResult

What is strange - this error happens only when I run my application on Kubernetes. When I run it on virtual machine or on my latop everything works OK

My application is using Spring Boot 2.2.1 and Spring Cloud Gateway and react

What I am doing wrong ?

My gradle dependencies

  implementation "com.github.vladimir-bukhtoyarov:bucket4j-core:6.2.0"
   implementation "com.github.vladimir-bukhtoyarov:bucket4j-jcache:6.2.0"
   implementation "com.github.vladimir-bukhtoyarov:bucket4j-hazelcast:6.2.0"
   implementation "javax.cache:cache-api:1.0.0"
   implementation "com.hazelcast:hazelcast:4.2"
   implementation "com.hazelcast:hazelcast-eureka-one:2.0.1"

My Hazelcast configuration code

 ```
    Config config = new Config();
   config.getNetworkConfig().getJoin().getTcpIpConfig().setEnabled(true);
        config.getNetworkConfig().getJoin().getMulticastConfig().setEnabled(false);
        config.getNetworkConfig().getJoin().getAwsConfig().setEnabled(false);
        config.getNetworkConfig().getJoin().getEurekaConfig().setEnabled(false);
        config.getNetworkConfig().getJoin().getKubernetesConfig().setEnabled(false);

        if (eurekaClientOption.isPresent()) {
            EurekaClient eurekaClient = eurekaClientOption.get();
            String appName = eurekaClient.getApplicationInfoManager().getEurekaInstanceConfig().getAppname();
            logger.info("hazelcast appName :" + appName);
            Application application = eurekaClient.getApplication(appName);
            if (application != null) {
                application.getInstancesAsIsFromEureka().forEach(
                        instanceInfo -> {
                            config.getNetworkConfig().getJoin().getTcpIpConfig().addMember(instanceInfo.getIPAddr() + ":" + port);
                            logger.info("hazelcast TcpIpConfig : {}",config.getNetworkConfig().getJoin().getTcpIpConfig());
                        }
                );
            } else {
                logger.error("hazelcast empty eureka application [{}]",appName);
            }
        } else {
            logger.error("hazelcast empty eurekaClient");
        }
        config.getNetworkConfig().setPort(port).setPortAutoIncrement(false);
        config.setLiteMember(false);
        logger.info("hazelcast config {}",config);
    config.setProperty("hazelcast.phone.home.enabled", "false");
    config.setProperty("hazelcast.initial.min.cluster.size","1");
    config.setProperty("hazelcast.socket.server.bind.any", "false");
   HazelcastInstance hazelcastInstance = new HazelcastInstanceFactory(config).getHazelcastInstance();

My bucket4j code

    ```
    if (hazelcastInstance != null) {
            IMap map = hazelcastInstance.getMap("bucket");
            int initialCapacity = (throttlingRule.getBandwith() + throttlingRule.getOverdraftBandwith());
            this.bucket = Bucket4j.extension(Hazelcast.class)
                    .builder()
                    .addLimit(Bandwidth.classic(initialCapacity, Refill.smooth(throttlingRule.getBandwith(), Duration.of(throttlingRule.getDuration(), throttlingRule.getDurationUnit()))))
                    .build(map, key, RecoveryStrategy.RECONSTRUCT);

            VerboseResult<EstimationProbe> verboseResult = bucket.asVerbose().estimateAbilityToConsume(1);
            BucketConfiguration bucketConfiguration = verboseResult.getConfiguration();
            logger.info("bucket for rule [{}] and key [{}] configuration [{}] availableTokens [{}]",throttlingRule,key,bucketConfiguration,bucket.getAvailableTokens());
        } else  {
            logger.warn("hazelcastInstance == null");
        }

  public ConsumptionProbe throttleProbe() {
       assert bucket != null;
       CompletableFuture<ConsumptionProbe> probe = bucket.asAsync().tryConsumeAndReturnRemaining(1);
       return probe.join();
   }

Exception

j.l.ClassNotFoundException: io.github.bucket4j.grid.CommandResult
	at j.i.l.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
	at j.i.l.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
	at j.lang.ClassLoader.loadClass(ClassLoader.java:522)
	at c.h.i.n.ClassLoaderUtil.tryLoadClass(ClassLoaderUtil.java:289)
	at c.h.i.n.ClassLoaderUtil.loadClass(ClassLoaderUtil.java:249)
	at c.h.i.n.IOUtil$ClassLoaderAwareObjectInputStream.resolveClass(IOUtil.java:910)
	at j.i.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1995)
	at j.i.ObjectInputStream.readClassDesc(ObjectInputStream.java:1862)
	at j.i.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2169)
	at j.i.ObjectInputStream.readObject0(ObjectInputStream.java:1679)
	at j.i.ObjectInputStream.readObject(ObjectInputStream.java:493)
	at j.i.ObjectInputStream.readObject(ObjectInputStream.java:451)
	at c.h.i.s.i.d.JavaDefaultSerializers$JavaSerializer.read(JavaDefaultSerializers.java:86)
	... 11 common frames omitted
Wrapped by: c.h.n.s.HazelcastSerializationException: java.lang.ClassNotFoundException: io.github.bucket4j.grid.CommandResult
	at c.h.i.s.i.d.JavaDefaultSerializers$JavaSerializer.read(JavaDefaultSerializers.java:90)
	at c.h.i.s.i.d.JavaDefaultSerializers$JavaSerializer.read(JavaDefaultSerializers.java:79)
	at c.h.i.s.i.StreamSerializerAdapter.read(StreamSerializerAdapter.java:44)
	at c.h.i.s.i.AbstractSerializationService.toObject(AbstractSerializationService.java:208)
	at c.h.s.i.DelegatingCompletableFuture$DeserializingFunction.apply(DelegatingCompletableFuture.java:518)
	at c.h.s.i.AbstractInvocationFuture.lambda$unblockCompose$5(AbstractInvocationFuture.java:953)
	at j.u.c.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1426)
	at j.u.c.ForkJoinTask.doExec(ForkJoinTask.java:290)
	at j.u.c.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
	at j.u.c.ForkJoinPool.scan(ForkJoinPool.java:1656)
	at j.u.c.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
	at j.u.c.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
Wrapped by: j.u.c.CompletionException: com.hazelcast.nio.serialization.HazelcastSerializationException: java.lang.ClassNotFoundException: io.github.bucket4j.grid.CommandResult
	at j.u.c.CompletableFuture.encodeThrowable(CompletableFuture.java:331)
	at j.u.c.CompletableFuture.completeThrowable(CompletableFuture.java:346)
	at j.u.c.CompletableFuture$UniApply.tryFire(CompletableFuture.java:632)
	at j.u.c.CompletableFuture.unipush(CompletableFuture.java:589)
	at j.u.c.CompletableFuture.uniApplyStage(CompletableFuture.java:660)
	at j.u.c.CompletableFuture.thenApplyAsync(CompletableFuture.java:2104)
	at c.h.s.i.InternalCompletableFuture.thenApply(InternalCompletableFuture.java:78)
	at i.g.b.AbstractBucket$1.tryConsumeAndReturnRemaining(AbstractBucket.java:147)
	at p.b.g.t.ThrottlingRuleBucket.throttleProbe(ThrottlingRuleBucket.java:64)

[vladimir-bukhtoyarov]

Hello @Jacuo,

Exception explicitly describes the problem, you need to be sure that Bucket4j related jars present on all Hazelcast nodes.

[Jacuo]

Hello @Jacuo,

Exception explicitly describes the problem, you need to be sure that Bucket4j related jars present on all Hazelcast nodes.

In fact as you can see all my nodes are embedded nodes ( the same application started on K8S cluster, which forms Hazelcast cluter, there is no "external" cluster ) - so on all nodes there is bucket4j-core . At least it worked this way in Hazelcast 3.x.

[Jacuo]

When I call bucket.getAvailableTokens() - it works.
But bucket.asAsync().tryConsumeAndReturnRemaining(1) causes exception
Maybe problem is with bucket.asAsync() ?

[vladimir-bukhtoyarov]

I've just retested Bucket4j 6.2 with Hazelcast version provided in your first message com.hazelcast:hazelcast:4.2 and I did not find any problems with Serialization(includes async operations).

It will be better just to debug classloading remotely

Just use breakpoint expression to minimize interruptions

[Jacuo]

Thanks @vladimir-bukhtoyarov I will try to debug. Problem is, that doesn`t happen, when I run it on my laptop :-)
Anyway , when I have changed from CompletableFuture probe = bucket.asAsync().tryConsumeAndReturnRemaining(1); to CompletableFuture probe = bucket.tryConsumeAndReturnRemaining(1); error disappeared

[vladimir-bukhtoyarov]

@vladimir-bukhtoyarov

Please look to the #186 (comment) maybe you have a problem with Dockerfile

[jigarcaptain]

I have the same issue. It works when I run application in IDE, when I run the application in docker-compose, it fails. My hazelcast member docker container has all the jars in classpath. Behavior changed when I run my application directly on my host vs inside of docker.

I just saw above reply. I am gonna try the file permission suggestion and I will report back.

[jigarcaptain]

I check the jar permission and they were correct to begin with, I even try 777 permission but I face the same error.
Hazelcast member in docker works with bucket4j client in my laptop, but does not work with same bucket4j client inside docker container. I have a custom EntryProcessor on client too, which works fine BTW.

[jigarcaptain]

Here is the Hazelcast member docker container

FROM hazelcast/hazelcast:5.0

ENV BUCKET4J_VERSION 6.3.0
ENV JCACHE_VERSION 1.1.1
ENV HZ_VERSION 5.0
ENV HZ_LIB_PATH $HZ_HOME/bin/user-lib

USER root
# Download jcache and bucket4j jars from maven repo.
ADD https://repo1.maven.org/maven2/javax/cache/cache-api/$JCACHE_VERSION/cache-api-$JCACHE_VERSION.jar $HZ_LIB_PATH
ADD https://repo1.maven.org/maven2/com/github/vladimir-bukhtoyarov/bucket4j-core/$BUCKET4J_VERSION/bucket4j-core-$BUCKET4J_VERSION.jar $HZ_LIB_PATH
ADD https://repo1.maven.org/maven2/com/github/vladimir-bukhtoyarov/bucket4j-jcache/$BUCKET4J_VERSION/bucket4j-jcache-$BUCKET4J_VERSION.jar $HZ_LIB_PATH
ADD https://repo1.maven.org/maven2/com/github/vladimir-bukhtoyarov/bucket4j-hazelcast/$BUCKET4J_VERSION/bucket4j-hazelcast-$BUCKET4J_VERSION.jar $HZ_LIB_PATH
RUN chmod 644 $HZ_LIB_PATH/*.jar
USER hazelcast

EXPOSE 5701

And client configuration

        final ClientUserCodeDeploymentConfig userCodeDeploymentConfig = new ClientUserCodeDeploymentConfig();
        userCodeDeploymentConfig.addClass(EntryProcessorOne.class.getName());
        userCodeDeploymentConfig.addClass(EntryProcessorTwo.class.getName());
        userCodeDeploymentConfig.setEnabled(true);

        final ClientNetworkConfig networkConfig = new ClientNetworkConfig();
        networkConfig.setAddresses(addresses);
        return new ClientConfig()
                .setClusterName(clusterName)
                .setInstanceName(String.format("%s-%s", clusterName, UUID.randomUUID()))
                .setUserCodeDeploymentConfig(userCodeDeploymentConfig)
                .setNetworkConfig(networkConfig);

[jigarcaptain]

I solved my issue with help of this hazelcast/hazelcast#11217 (comment)
Looks like this is a bug in Hazelcast. Issue was with class loading on client and not on member side.

Following change fixed the issue for me.

        return new ClientConfig()
                .setClassLoader(CommandResult.class.getClassLoader())        <------ THIS LINE HERE
                .setClusterName(clusterName)
                .setInstanceName(String.format("%s-%s", clusterName, UUID.randomUUID()))
                .setUserCodeDeploymentConfig(userCodeDeploymentConfig)
                .setNetworkConfig(networkConfig);

[vladimir-bukhtoyarov]

Hello @jigarcaptain

Thank you for your investigation. I do not use SpringBoot, but I know that SpringBoot is a popular framework, so it looks that hint with ClientConfig#setClassLoader needs to be mentioned in the bucket4j-hazelcast documentation. I will do this at weekend.

[vladimir-bukhtoyarov]

@MarcGiffing @marcosbarbero

Hello guys. Looks that your integrations with Bucket4j can have the same problems.

[marcosbarbero]

Thanks for letting me know!