Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"PortableFactory[-22] is already registered" error w/ Spring Boot 1.4.2 & Hazelcast 3.7.x #10438

Closed
mmoayyed opened this issue Apr 26, 2017 · 19 comments

Comments

@mmoayyed
Copy link

@mmoayyed mmoayyed commented Apr 26, 2017

Problem

This issue is almost identical to #1826 except that I might have a bit more information:

The stacktrace is very similar:

Caused by: java.lang.IllegalArgumentException: PortableFactory[-22] is already registered! com.hazelcast.concurrent.countdownlatch.CountDownLatchPortableHook$1@1ed6352 -> 
com.hazelcast.concurrent.countdownlatch.client.CountDownLatchPortableHook$1@13dc0e
at com.hazelcast.nio.serialization.PortableHookLoader.register(PortableHookLoader.java:84)
at com.hazelcast.nio.serialization.PortableHookLoader.load(PortableHookLoader.java:51)
at com.hazelcast.nio.serialization.PortableHookLoader.(PortableHookLoader.java:41)
at com.hazelcast.nio.serialization.SerializationServiceImpl.(SerializationServiceImpl.java:85)
at com.hazelcast.nio.serialization.SerializationServiceBuilder.build(SerializationServiceBuilder.java:174)
at com.hazelcast.instance.Node.(Node.java:130)
at com.hazelcast.instance.HazelcastInstanceImpl.(HazelcastInstanceImpl.java:92)
at com.hazelcast.instance.HazelcastInstanceFactory.newHazelcastInstance(HazelcastInstanceFactory.java:92)
at com.hazelcast.instance.HazelcastInstanceFactory.newHazelcastInstance(HazelcastInstanceFactory.java:72)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:166)

Reproduce

The issue was originally reported here:
https://github.com/apereo/cas/issues/2559

A sample project that demonstrates the issue is also available here:
https://github.com/apereo/cas/files/952931/cas505.zip

To duplicate, you want to mvn clean package and then run the generated war with java -jar target/cas.war.

The main difference between this issue and #1826 besides version numbers is that this time the application at fault is an executable war. The issue does not present itself with using bootRun sort of tasks or when deploying the application inside an external Tomcat instance.

Diagnosis

Tracking this down, I realized that Hazelcast's ServiceLoader component deals with multiple ClassLoaders. See the screenshot below:

image

As you can see, there are two classloaders here: one that is used to launch the WAR, and another that deals with the embedded tomcat web application. They both attempt to load the same sort of resource/service and as a result, run into conflicts. The URIs used seem identical except that protocol perhaps.

This is way too deep for me to go any further, but perhaps, could hazelcast ignore duplicate resources/services such as c.h.i.NodeExtension if they have already been loaded?

@jerrinot
Copy link
Contributor

@jerrinot jerrinot commented Apr 26, 2017

hello @mmoayyed,

thanks for an excellent bug report. I can confirm I can reproduce the issue. Let's see what we can do about it.

@mmoayyed
Copy link
Author

@mmoayyed mmoayyed commented Apr 26, 2017

Very good. Appreciate your support. If there is anything I can do to assist or test, by all means, ping.

@jerrinot
Copy link
Contributor

@jerrinot jerrinot commented Apr 26, 2017

Your analysis was spot-on.

I think I understand what is going - Hazelcast loads resources embedded in the JAR.
For example META-INF/services/com.hazelcast.instance.NodeExtension.

The selectClassLoaders() picks classloaders:

  1. It will pick TCCL when it is set. In this case it's set to the TomcatEmbeddedWebappClassLoader

  2. then it will pick the "core" classloader - the same classloader which is used to load the ServiceLoader class. unless it's the same at the TCCL classloader. In this case it's set to the LaunchedURLClassLoader

Then it will use both classloaders to find resources. The problem is the TomcatEmbeddedWebappClassLoader and ``LaunchedURLClassLoader` will both load resources from the same physical JAR, but they will present it under 2 distinct URL!

Hence the same physical resources from the same JAR appears as 2 distinct resources to Hazelcast.
In theory we could indeed skip resources such as c.h.i.NodeExtension, but this smells like asking for hard-to-troubleshoot issues. Let's see if we can do better.

Ideally Spring Boot should not present the same resources under 2 unique URLs.

@mmoayyed
Copy link
Author

@mmoayyed mmoayyed commented Apr 26, 2017

You are right on all counts. That's exactly what I witnessed as well.

Some additional points:

  • In order to temporarily "fix" the solution I had to reset the TCCL to something that would remove the need to pick TomcatEmbeddedWebappClassLoader as well, but that seemed like a very scary thing to do.
  • Interestingly enough, switching to more recent versions of CAS that are based off of Spring Boot 1.5.3, (Note the original is based on 1.4.2; HZ version remains the same here), I cannot duplicate the issue! It's possible that Boot has changed the way the classloaders work, or perhaps CAS has reorganized internally somehow. (Very much doubt the latter).

More context:

One thing that is also peculiar about the sample project above is that it's not exactly itself an executable war. It wraps itself around an already executable war, using a Maven overlay. The overlay project as you may have also noticed does the following more or less:

  1. Downloads a bootified-embeds-tomcat-in-it prebuilt CAS web application binary (1- cas-server-webapp.5.0.0.war) from central
  2. Unpacks, injects stuff into it if needed from the local project and...
  3. Follows through the Maven overlay process to package everything up (2- `cas.war).

All standard Maven stuff. Now, I am not 100% sure if that whole process messes up classloaders in some way, but given (2) absolutely does nothing with (1) in this case, the final result should be identical either way. To further confirm this theory, I abandoned the sample project above to remove all weirdness caused potentially by the Maven overlay process and the war plugin and simply ran the cas web application (built from source; not downloaded from central this time) with a java -jar command. Same error. So I am inclined to not blame Maven this time around :)

We have been running CAS and Hazelcast in production for many years for many deployments. It's always been a pleasure. Thank you for your follow through!

jerrinot added a commit to jerrinot/hazelcast that referenced this issue Apr 26, 2017
jerrinot added a commit to jerrinot/hazelcast that referenced this issue Apr 26, 2017
@jerrinot
Copy link
Contributor

@jerrinot jerrinot commented Apr 26, 2017

@mmoayyed: Are you a Spring Boot expert? Could you create a simple Hello-World style project loading a resource from a JAR and demonstrating the same physical resource is available under 2 distinct URLs?

I think that would help to pin-point the real root cause and perhaps a reason to open a new issue in the Spring Boot project.

I'm playing with #10444 - it does the job, but I am not sure if that's the right way to tackle the issue.

@mmoayyed
Copy link
Author

@mmoayyed mmoayyed commented Apr 26, 2017

No expert, but I can find my way around it :) I'll see about that Hello-World sample, sure. Stand by please.

jerrinot added a commit to jerrinot/hazelcast that referenced this issue Apr 27, 2017
@jerrinot
Copy link
Contributor

@jerrinot jerrinot commented Apr 27, 2017

@mmoayyed: I managed to create a reproducer and opened a bug in the Spring Boot project: spring-projects/spring-boot#9014

@mmoayyed
Copy link
Author

@mmoayyed mmoayyed commented Apr 27, 2017

Thank you for beating me to this! I'll set a watch on that issue as well.

@majidvodworks
Copy link

@majidvodworks majidvodworks commented Jun 20, 2017

I am also facing similar issue, I want to upgrade to hazelcast 3. 8 but getting following exception when starting app. I am using spring boot 1.4.4.

Caused by: java.lang.IllegalArgumentException: PortableFactory[-10] is already registered! com.hazelcast.map.impl.client.MapPortableHook$MapPortableFactory@671f545b -> com.hazelcast.map.impl.MapPortableHook$1@c335b9

@mmedenjak
Copy link
Contributor

@mmedenjak mmedenjak commented Aug 3, 2017

@tombujok @jerrinot the fix in spring-boot has been merged. Can we close this issue as well?

@jerrinot
Copy link
Contributor

@jerrinot jerrinot commented Aug 22, 2017

@mmedenjak: you are right. there is a fix in Spring Boot + we added a workaround on our side-as well.

@jerrinot jerrinot closed this Aug 22, 2017
@davisd123
Copy link

@davisd123 davisd123 commented Sep 6, 2017

Hi, has this been fixed in 3.8.5 or is it due out in 3.9?

I am having similar issue:
Failed to instantiate [com.hazelcast.core.HazelcastInstance]: Factory method 'hazelcastInstance' threw exception; nested exception is java.lang.IllegalArgumentException: PortableFactory[-23] is already registered! com.hazelcast.mapreduce.impl.MapReducePortableHook$1@325e8f98 -> com.hazelcast.mapreduce.impl.MapReducePortableHook$1@8e7f7a6

Hazelcast 3.8.4/3.8.5
Springboot 1.5.6
War deployment, using Payara 4.1.2.173

@mmedenjak
Copy link
Contributor

@mmedenjak mmedenjak commented Sep 6, 2017

@davisd123 looks like the fix in Spring boot was targeted for 2.0.0.M3 and the fix on our side was targeted for 3.9

@davisd123
Copy link

@davisd123 davisd123 commented Sep 7, 2017

@mmedenjak so that's 3.9 or will I find the fix in 3.9-EA? Thanks...

@mmedenjak
Copy link
Contributor

@mmedenjak mmedenjak commented Sep 7, 2017

@davisd123 that would be 3.9-GA (or 3.9 as you say). Until then you can try out the SNAPSHOT release to see if your issue is fixed : https://github.com/hazelcast/hazelcast#snapshot-releases

@davisd123
Copy link

@davisd123 davisd123 commented Sep 11, 2017

@mmedenjak hi - quick update - can't confirm if your suggestion is working or not (Springboot 2.0.0.M3 + Hazelcast 3.9-SNAPSHOT. The version of Payara (4.1.2.172) I'm using has Hazelcast 3.8.0 built in, and that looks like it's getting picked up first by the classloader.

If I can't sort this I'll switch to Tomcat and can then probably verify your solution... :)

@davisd123
Copy link

@davisd123 davisd123 commented Sep 12, 2017

@mmedenjak ok, I can confirm that the issue goes away with Hazelcast 3.9-SNAPSHOT and Springboot 2.0.0.M3.

Had a classloader issue with Payara - to fix with Glassfish/Payara, refer to the following guide to set the class-loader delegate="false": https://docs.payara.fish/documentation/extended-documentation/classloading.html

@jerrinot
Copy link
Contributor

@jerrinot jerrinot commented Sep 12, 2017

@davisd123: thanks for reporting back, much appreciated!

@radut
Copy link

@radut radut commented Dec 3, 2017

I confirm that is working with spring boot 1.5.9.RELEASE and Hazelcast version 3.9.1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

6 participants
You can’t perform that action at this time.