ClassNotFoundException when using an JCache EntryProcessor PAYARA-1257

[OndroMih]

This issue is created from GitHub gist by @ivannov:
https://gist.github.com/ivannov/906096dcff2100ccd5b2ef11f97362dd

More detailed information on Java EE Guardian mailing list: https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/javaee-guardians/8QxcgzGdt1o/dMU0wqh0LgAJ

Summary:

We implemented our simple entry processor by the JCache spec.
cache.invoke() with the entry processor fails on java.lang.ClassNotFoundException: bg.jug.guestbook.entities.Comment

[fturizo]

@OndrejM, any progress on this issue?

[OndroMih]

No progress, still to reproduce and confirm if it's a bug.

[ivannov]

@OndrejM as it was reported as a result of the Bulgarian JUG hackathon on JCache, here is a sample project:
https://github.com/bgjug/jcache-workshop

You can find there an application where it is reproduced and under the docs directory are the (pretty lengthy) instructions to get there

[jerrinot]

This will be indeed tricky to solve.

Hazelcast is part of Payara and it is not loaded by the same classloader as a webapp. When you do map.get() then I believe Payara set TCCL to the application classloader -> Hazelcast finds your application class when deserializing the result -> everything appears to work.

However when you invoke an entry processor then the receiving Hazelcast member is not deserializing it on user-thread -> TCCL is not set -> ClassNotFoundException

For Payara Micro this could be solved by setting Hazelcast classloader to the same classloader as the webapp uses, but I am not sure how to solve this in general.

[smillidge]

It's possible we could write a classloader that could reach into the different deployed applications to look for classes and add that to Hazelcast. We did an experiment by creating one and bootstrapping Hazelcast with that classloader as its TCCL however that caused service loading issues in Hazelcast. We haven't tried the Config class. Another alternative we've been told may work is a custom Hazelcast Serializer.

[jerrinot]

Of course, I would never write something like this

(I double-checked the commit is not signed so I can still maintain plausible deniability :-D)

[smillidge]

did it work though?

[jerrinot]

It did work. I deployed the demo application @ivannov linked + added a simple entry processor.

Then I started a standalone application with a Hazelcast client and invoked the entry processor from the client. The entry processor (packaged inside my WAR) was executed on a Payara member.

I talked to @vbekiaris about this and it appears a real solution would require 1st class multi-tenancy support inside Hazelcast. 1 app = 1 Hazelcast tenant. Otherwise there is really no good way to link a serialized blob to an application context. Unless you want to start a cluster per application.

[lprimak]

Has anyone tried to put hazelcast into applibs classloader? That may be a good workaround until something is devised by Payara. I have some ideas on automating this.

[lprimak]

I have an idea:
appRegistry can be used to cycle & delegate through all app class loaders (i.e. delegated class loader) and it's used for the Hazelcast serialize classloader. This would be simple and builds on the work I've already done

[jerrinot]

@lprimak: you mean something like this?

one problem with this approach is that it breaks classloader isolation between applications. imagine you have different versions of a class packaged in different applications.

[smillidge]

we could potentially put an indication into the serialization stream with a custom serializer that indicated the application the class is from.

App registry with a custom serializer should work

[lprimak]

Yes, or something similar

[lprimak]

@ivannov I am working on Payara fixes so your workarounds are no longer necessary

[lprimak]

I am excited to report that the application now works correctly using the custom serializer.
Cleanup / productionalization of this will take some more doing, but the proof of concept works.

[lprimak]

Guestbook app that works with these fixes:
jcache-workshop.zip

[OndroMih]

Hi @ivannov,
This issue should be fixed now in master and will be available in version 172. If you have some time, you can test it even now with the latest nightly build (http://www.payara.fish/upstream_builds)

[lprimak]

Relates to hazelcast/hazelcast#10875

[lprimak]

Should work since 2018