Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Hazelcast 3.5.3 conflicts with FUSE 6.1 #6821

Closed
thomasmodeneis opened this issue Nov 23, 2015 · 48 comments

Comments

Projects
None yet
3 participants
@thomasmodeneis
Copy link

commented Nov 23, 2015

Hi,

It all started two weeks ago when the team decided it was a good idea to integrate Hazelcast into our ESB with JBOSS Fuse 6.1.

After few days playing with maven dependency (import/export) packages, I managed to get a version that seems to be working, however few warns in the logs are concerning me.

I'm not sure how to look for this *(old version of Hazelcast), I've been digging around but can't manage to figure out where this may be, so I decided to come asking for help.
Please apologize me if this is the wrong place, but I can't manage to find any other place to look for help.

2015-11-23 10:10:36,448 | WARN  | .MulticastThread | MulticastService                 | ?                                   ? | 274 - com.hazelcast-endpoint - 1.0.0.RC1-SNAPSHOT | [10.0.2.15]:5701 [dev] [3.5.3] Received data format is invalid. (An old version of Hazelcast may be running here.)
com.hazelcast.nio.serialization.HazelcastSerializationException: java.lang.IllegalStateException: Bundle is uninstalled
    at com.hazelcast.nio.serialization.SerializationServiceImpl.handleException(SerializationServiceImpl.java:380)[274:com.hazelcast-endpoint:1.0.0.RC1-SNAPSHOT]
JBossFuse:karaf@root> list
START LEVEL 100 , List Threshold: 50
   ID   State         Blueprint      Spring    Level  Name
[  67] [Active     ] [            ] [       ] [   50] Fabric8 :: ConfigAdmin Bridge (1.0.0.redhat-412)
[  68] [Active     ] [Created     ] [       ] [   50] Fabric8 :: Karaf Commands (1.0.0.redhat-412)
[  69] [Active     ] [Created     ] [       ] [   50] JBoss Fuse :: ESB :: Commands (6.1.1.redhat-412)
[  84] [Active     ] [            ] [       ] [   60] Guava: Google Core Libraries for Java (15.0.0)
[  85] [Active     ] [            ] [       ] [   60] Fabric8 :: Groups (1.0.0.redhat-412)
[  86] [Active     ] [            ] [       ] [   60] Fabric8 :: Git (1.0.0.redhat-412)
[  87] [Active     ] [            ] [       ] [   60] Fuse Common :: Util (1.0.0.redhat-412)
[  88] [Active     ] [Created     ] [       ] [   60] Fabric8 :: JAAS (1.0.0.redhat-412)
[ 104] [Active     ] [            ] [       ] [   60] Fabric8 :: Git :: Server (1.0.0.redhat-412)
[ 105] [Active     ] [            ] [       ] [   60] Fabric8 :: Redirect (1.0.0.redhat-412)
[ 106] [Active     ] [            ] [       ] [   60] Fabric8 :: Extender Listener (1.0.0.redhat-412)
[ 107] [Active     ] [Created     ] [       ] [   60] Fuse Application Bundles :: OSGi (1.0.0.redhat-412)
[ 108] [Active     ] [            ] [       ] [   60] OPS4J Pax Swissbox :: Optional JCL (1.6.0)
[ 109] [Active     ] [            ] [       ] [   60] OPS4J Base - Monitors (1.4.0)
[ 110] [Active     ] [            ] [       ] [   60] OPS4J Base - Util - Property (1.4.0)
[ 111] [Active     ] [            ] [       ] [   60] OPS4J Pax Swissbox :: Property (1.6.0)
[ 112] [Active     ] [            ] [       ] [   60] OPS4J Base - Net (1.4.0)
[ 113] [Active     ] [            ] [       ] [   60] Fabric8 :: Maven Proxy (1.0.0.redhat-412)
[ 114] [Active     ] [Created     ] [       ] [   60] Fuse Patch :: Commands (1.0.0.redhat-412)
[ 115] [Active     ] [Created     ] [       ] [   60] Fuse Patch :: Core (1.0.0.redhat-412)
[ 127] [Active     ] [            ] [       ] [   50] geronimo-j2ee-management_1.1_spec (1.0.1)
[ 128] [Active     ] [            ] [       ] [   50] geronimo-jms_1.1_spec (1.1.1)
[ 129] [Active     ] [            ] [       ] [   50] Apache ServiceMix :: Bundles :: jaxb-impl (2.2.1.1_2)
[ 130] [Active     ] [            ] [       ] [   50] JAXB2 Basics - Runtime (0.6.4)
[ 131] [Active     ] [            ] [       ] [   50] Commons Pool (1.6.0)
[ 132] [Active     ] [Created     ] [       ] [   50] activemq-osgi (5.9.0.redhat-611412)
[ 133] [Active     ] [            ] [       ] [   50] Commons Net (3.3.0)
[ 134] [Active     ] [            ] [       ] [   50] Apache XBean :: Spring (3.15.0)
[ 135] [Active     ] [            ] [       ] [   50] jettison (1.3.5)
[ 136] [Active     ] [            ] [       ] [   50] Apache ServiceMix :: Bundles :: xpp3 (1.1.0.4c_5)
[ 137] [Active     ] [            ] [       ] [   50] Apache ServiceMix :: Bundles :: xstream (1.4.7.1)
[ 138] [Active     ] [            ] [       ] [   50] activeio-core (3.1.4)
[ 139] [Active     ] [            ] [       ] [   50] Joda-Time (1.6.2)
[ 140] [Active     ] [Created     ] [       ] [   50] activemq-karaf (5.9.0.redhat-611412)
[ 141] [Active     ] [            ] [       ] [   60] Apache XBean :: Classloader (3.15.0)
[ 142] [Active     ] [            ] [       ] [   60] JBoss A-MQ Fabric (6.1.1.redhat-412)
[ 143] [Active     ] [Created     ] [       ] [   50] camel-karaf-commands (2.12.0.redhat-611412)
[ 144] [Active     ] [            ] [       ] [   50] camel-core (2.12.0.redhat-611412)
[ 150] [Active     ] [            ] [       ] [   50] camel-spring (2.12.0.redhat-611412)
[ 151] [Active     ] [Created     ] [       ] [   50] camel-blueprint (2.12.0.redhat-611412)
[ 204] [Active     ] [            ] [       ] [   50] Apache CXF Compatibility Bundle Jar (2.7.0.redhat-611412)
[ 205] [Active     ] [Created     ] [       ] [   50] camel-cxf (2.12.0.redhat-611412)
[ 206] [Active     ] [Created     ] [       ] [   50] camel-cxf-transport (2.12.0.redhat-611412)
[ 207] [Active     ] [            ] [       ] [   50] camel-jms (2.12.0.redhat-611412)
[ 208] [Active     ] [            ] [       ] [   60] activemq-camel (5.9.0.redhat-611412)
[ 209] [Active     ] [            ] [       ] [   60] JBoss A-MQ Fabric Camel Component (6.1.1.redhat-412)
[ 210] [Active     ] [            ] [       ] [   50] Apache ServiceMix :: Bundles :: commons-csv (1.0.0.r706899_5)
[ 211] [Active     ] [            ] [       ] [   50] camel-csv (2.12.0.redhat-611412)
[ 212] [Active     ] [            ] [       ] [   50] camel-ftp (2.12.0.redhat-611412)
[ 213] [Active     ] [            ] [       ] [   50] camel-bindy (2.12.0.redhat-611412)
[ 214] [Active     ] [            ] [       ] [   50] camel-jdbc (2.12.0.redhat-611412)
[ 215] [Active     ] [            ] [       ] [   50] Commons Exec (1.1)
[ 216] [Active     ] [            ] [       ] [   50] Apache Commons IO Bundle (1.4)
[ 217] [Active     ] [            ] [       ] [   50] camel-exec (2.12.0.redhat-611412)
[ 218] [Active     ] [            ] [       ] [   50] camel-jasypt (2.12.0.redhat-611412)
[ 219] [Active     ] [            ] [       ] [   50] Apache ServiceMix :: Bundles :: Saxon-HE (9.5.1.2_1)
[ 220] [Active     ] [            ] [       ] [   50] camel-saxon (2.12.0.redhat-611412)
[ 221] [Active     ] [            ] [       ] [   50] Apache ServiceMix :: Bundles :: snmp4j (2.2.3.1)
[ 222] [Active     ] [            ] [       ] [   50] camel-snmp (2.12.0.redhat-611412)
[ 223] [Active     ] [            ] [       ] [   50] Apache ServiceMix :: Bundles :: ognl (3.0.8.1)
[ 224] [Active     ] [            ] [       ] [   50] Apache ServiceMix :: Bundles :: javassist (3.12.1.GA_3)
[ 225] [Active     ] [            ] [       ] [   50] camel-ognl (2.12.0.redhat-611412)
[ 226] [Active     ] [            ] [       ] [   50] camel-routebox (2.12.0.redhat-611412)
[ 227] [Active     ] [            ] [       ] [   50] Apache ServiceMix :: Specs :: Scripting API 1.0 (2.3.0.redhat-611412)
[ 228] [Active     ] [            ] [       ] [   50] camel-script (2.12.0.redhat-611412)
[ 229] [Active     ] [            ] [       ] [   50] Groovy Runtime (2.1.9)
[ 231] [Active     ] [            ] [       ] [   50] camel-spring-javaconfig (2.12.0.redhat-611412)
[ 232] [Active     ] [            ] [       ] [   50] camel-jaxb (2.12.0.redhat-611412)
[ 233] [Active     ] [            ] [       ] [   50] Apache ServiceMix :: Bundles :: commons-httpclient (3.1.0.7)
[ 234] [Active     ] [            ] [       ] [   50] camel-http (2.12.0.redhat-611412)
[ 235] [Active     ] [            ] [       ] [   50] camel-jetty (2.12.0.redhat-611412)
[ 236] [Active     ] [            ] [       ] [   50] camel-jmx (2.12.0.redhat-611412)
[ 237] [Active     ] [            ] [       ] [   50] camel-mail (2.12.0.redhat-611412)
[ 238] [Active     ] [            ] [       ] [   50] camel-paxlogging (2.12.0.redhat-611412)
[ 239] [Active     ] [            ] [       ] [   50] camel-rmi (2.12.0.redhat-611412)
[ 246] [Active     ] [            ] [       ] [   60] hawtio :: hawtio-osgi-jmx (1.2.0.redhat-412)
[ 247] [Active     ] [Created     ] [       ] [   60] hawtio :: hawtio-json-schema-mbean (1.2.0.redhat-412)
[ 248] [Active     ] [            ] [       ] [   60] hawtio :: hawtio-web (1.2.0.redhat-412)
[ 249] [Active     ] [            ] [       ] [   60] JLine (2.11.0)
[ 250] [Active     ] [Created     ] [       ] [   60] hawtio :: Karaf terminal plugin (1.2.0.redhat-412)
[ 251] [Active     ] [Created     ] [       ] [   60] hawtio :: hawtio-maven-indexer (1.2.0.redhat-412)
[ 252] [Active     ] [            ] [       ] [   50] Apache HttpClient OSGi bundle (4.2.5)
[ 253] [Active     ] [            ] [       ] [   50] Apache HttpCore OSGi bundle (4.2.4)
[ 254] [Active     ] [            ] [       ] [   50] camel-http4 (2.12.0.redhat-611412)
[ 255] [Active     ] [            ] [       ] [   50] Apache ServiceMix :: Bundles :: ant (1.7.0.6)
[ 256] [Active     ] [            ] [       ] [   50] camel-groovy (2.12.0.redhat-611412)
[ 257] [Active     ] [            ] [       ] [   50] Apache ServiceMix :: Bundles :: aws-java-sdk (1.5.1.1)
[ 258] [Active     ] [            ] [       ] [   50] camel-aws (2.12.0.redhat-611412)
[ 279] [Active     ] [            ] [       ] [   50] Apache ServiceMix :: Bundles :: velocity (1.7.0.6)
[ 280] [Active     ] [            ] [       ] [   50] Apache ServiceMix :: Bundles :: jdom (1.1.0.4)
[ 281] [Active     ] [            ] [       ] [   50] Apache ServiceMix :: Bundles :: oro (2.0.8.6)
[ 282] [Active     ] [            ] [       ] [   50] camel-velocity (2.12.0.redhat-611412)
[ 288] [Active     ] [            ] [Started] [   60] hazelcast-endpoint (1.0.0.RC1-SNAPSHOT)
[ 289] [Active     ] [            ] [       ] [   60] esb-utils (7.1.0.RC2)
[ 290] [Active     ] [            ] [       ] [   60] Jackson-core (2.6.3)
[ 291] [Active     ] [            ] [       ] [   60] jackson-databind (2.6.3)
[ 292] [Active     ] [            ] [       ] [   60] Jackson-annotations (2.6.3)

@serkan-ozal serkan-ozal self-assigned this Nov 26, 2015

@serkan-ozal serkan-ozal added the PENDING label Nov 26, 2015

@serkan-ozal serkan-ozal added this to the 3.6 milestone Nov 26, 2015

@serkan-ozal serkan-ozal added Type: Defect and removed PENDING labels Nov 26, 2015

@serkan-ozal

This comment has been minimized.

Copy link
Contributor

commented Nov 27, 2015

Hi @thomasmodeneis
Are you sure that there is no running Hazelcast instance in your environment (inside OSGI environment as internally or externally from outside of OSGI environment)?

@thomasmodeneis

This comment has been minimized.

Copy link
Author

commented Nov 27, 2015

The artifact have been deployed into a clean installation of jboss-fuse-6.1.1.redhat-412, by that I mean no other service have ever been into this container, only the hazelcast-endpoint (1.0.0.RC1-SNAPSHOT).

This project was generated using some other poms that are private and adapted for the company I'm currently working on, and right now I'm working on to isolate completely this project from the other poms, I will be testing once again with this clean artifact and if the problem persists I will opensource the example into my git-hub in order to help you guys to easily identify the problem.
I will be posting here again as soon as I manage to complete this step.

Thank you for your attention.

@serkan-ozal

This comment has been minimized.

Copy link
Contributor

commented Nov 27, 2015

Hi @thomasmodeneis

Thanks for your response.

Besides 3.5.3, can you also try with the latest version (3.6-EA2)?

Regards.

@serkan-ozal

This comment has been minimized.

Copy link
Contributor

commented Dec 2, 2015

Hi @thomasmodeneis
Any update with this issue?

@serkan-ozal

This comment has been minimized.

Copy link
Contributor

commented Dec 7, 2015

@thomasmodeneis any update?

@thomasmodeneis

This comment has been minimized.

Copy link
Author

commented Dec 7, 2015

I just manage to get some time to work on it, hopefully will be done by the end of the day

@thomasmodeneis

This comment has been minimized.

Copy link
Author

commented Dec 7, 2015

Hi @serkan-ozal I've added the POC for the problem on github, please let me know if you need any clarifications. https://github.com/thomasmodeneis/hazelcast-fuse61

@serkan-ozal

This comment has been minimized.

Copy link
Contributor

commented Dec 7, 2015

Thanks @thomasmodeneis
I will investigate it

@serkan-ozal

This comment has been minimized.

Copy link
Contributor

commented Dec 8, 2015

Hi @thomasmodeneis

I have managed to run your sample (after struggling by adding other required features during a few hours to learn how Karaf works :) ) on Apache Karaf 2.4.3 without any exception.

Are you able to reproduce the issue with your sample project?
If yes, can you share your full Karaf logs and version?

Here is the log shows that Hazelcast feature is active:

[ 113] [Active     ] [            ] [Started] [   80] hazelcast-endpoint (1.0.0.RC1-SNAPSHOT)

Here is my karaf.log where there is no exception:

Dec 08, 2015 3:08:39 PM org.apache.karaf.main.SimpleFileLock lock
INFO: locking
2015-12-08 15:08:39,651 | INFO  | FelixStartLevel  | GuardProxyCatalog                | 25 - org.apache.karaf.service.guard - 2.4.3 | No compulsory roles for a karaf command without the ACL as its system property is not set: karaf.secured.command.compulsory.roles
2015-12-08 15:08:41,003 | INFO  | FelixStartLevel  | BlueprintExtender                | 14 - org.apache.aries.blueprint.core - 1.4.3 | No quiesce support is available, so blueprint components will not participate in quiesce operations
2015-12-08 15:08:41,604 | INFO  | FelixStartLevel  | SecuredCommandConfigTransformer  | 17 - org.apache.karaf.shell.console - 2.4.3 | Generating command ACL config org.apache.karaf.command.acl.features into service ACL configs [org.apache.karaf.service.acl.command.features.install, org.apache.karaf.service.acl.command.features.uninstall]
2015-12-08 15:08:41,609 | INFO  | FelixStartLevel  | SecuredCommandConfigTransformer  | 17 - org.apache.karaf.shell.console - 2.4.3 | Generating command ACL config org.apache.karaf.command.acl.config into service ACL configs [org.apache.karaf.service.acl.command.config.propappend, org.apache.karaf.service.acl.command.config.delete, org.apache.karaf.service.acl.command.config.propdel, org.apache.karaf.service.acl.command.config.cancel, org.apache.karaf.service.acl.command.config.propset, org.apache.karaf.service.acl.command.config.update, org.apache.karaf.service.acl.command.config.edit]
2015-12-08 15:08:41,621 | INFO  | FelixStartLevel  | SecuredCommandConfigTransformer  | 17 - org.apache.karaf.shell.console - 2.4.3 | Generating command ACL config org.apache.karaf.command.acl.shell into service ACL configs [org.apache.karaf.service.acl.command.shell.java, org.apache.karaf.service.acl.command.shell.edit, org.apache.karaf.service.acl.command.shell.exec, org.apache.karaf.service.acl.command.shell.new]
2015-12-08 15:08:41,625 | INFO  | FelixStartLevel  | SecuredCommandConfigTransformer  | 17 - org.apache.karaf.shell.console - 2.4.3 | Generating command ACL config org.apache.karaf.command.acl.jaas into service ACL configs [org.apache.karaf.service.acl.command.jaas.update]
2015-12-08 15:08:41,626 | INFO  | FelixStartLevel  | SecuredCommandConfigTransformer  | 17 - org.apache.karaf.shell.console - 2.4.3 | Generating command ACL config org.apache.karaf.command.acl.osgi into service ACL configs [org.apache.karaf.service.acl.command.osgi.watch, org.apache.karaf.service.acl.command.osgi.uninstall, org.apache.karaf.service.acl.command.osgi.shutdown, org.apache.karaf.service.acl.command.osgi.start-level, org.apache.karaf.service.acl.command.osgi.refresh, org.apache.karaf.service.acl.command.osgi.start, org.apache.karaf.service.acl.command.osgi.restart, org.apache.karaf.service.acl.command.osgi.install, org.apache.karaf.service.acl.command.osgi.update, org.apache.karaf.service.acl.command.osgi.stop]
2015-12-08 15:08:41,638 | INFO  | FelixStartLevel  | SecuredCommandConfigTransformer  | 17 - org.apache.karaf.shell.console - 2.4.3 | Generating command ACL config org.apache.karaf.command.acl.scope_bundle into service ACL configs [org.apache.karaf.service.acl.command.scope_bundle.shell, org.apache.karaf.service.acl.command.scope_bundle.admin, org.apache.karaf.service.acl.command.scope_bundle.packages, org.apache.karaf.service.acl.command.scope_bundle.ssh, org.apache.karaf.service.acl.command.scope_bundle.log, org.apache.karaf.service.acl.command.scope_bundle.jaas, org.apache.karaf.service.acl.command.scope_bundle.features, org.apache.karaf.service.acl.command.scope_bundle.osgi, org.apache.karaf.service.acl.command.scope_bundle.config]
2015-12-08 15:08:42,573 | INFO  | FelixStartLevel  | SecurityUtils                    | 30 - org.apache.sshd.core - 0.14.0 | BouncyCastle not registered, using the default JCE provider
2015-12-08 15:08:43,279 | INFO  | FelixStartLevel  | core                             | 34 - org.apache.aries.jmx.core - 1.1.3 | Starting JMX OSGi agent
2015-12-08 15:08:43,301 | INFO  | FelixStartLevel  | core                             | 34 - org.apache.aries.jmx.core - 1.1.3 | Registering MBean with ObjectName [osgi.compendium:service=cm,version=1.3,framework=org.apache.felix.framework,uuid=97de0fcc-449e-4063-8c94-bd817604f89c] for service with service.id [11]
2015-12-08 15:08:43,304 | INFO  | FelixStartLevel  | core                             | 34 - org.apache.aries.jmx.core - 1.1.3 | Registering org.osgi.jmx.framework.FrameworkMBean to MBeanServer com.sun.jmx.mbeanserver.JmxMBeanServer@725b1426 with name osgi.core:type=framework,version=1.7,framework=org.apache.felix.framework,uuid=97de0fcc-449e-4063-8c94-bd817604f89c
2015-12-08 15:08:43,304 | INFO  | FelixStartLevel  | core                             | 34 - org.apache.aries.jmx.core - 1.1.3 | Registering org.osgi.jmx.framework.ServiceStateMBean to MBeanServer com.sun.jmx.mbeanserver.JmxMBeanServer@725b1426 with name osgi.core:type=serviceState,version=1.7,framework=org.apache.felix.framework,uuid=97de0fcc-449e-4063-8c94-bd817604f89c
2015-12-08 15:08:43,306 | INFO  | FelixStartLevel  | core                             | 34 - org.apache.aries.jmx.core - 1.1.3 | Registering org.osgi.jmx.framework.PackageStateMBean to MBeanServer com.sun.jmx.mbeanserver.JmxMBeanServer@725b1426 with name osgi.core:type=packageState,version=1.5,framework=org.apache.felix.framework,uuid=97de0fcc-449e-4063-8c94-bd817604f89c
2015-12-08 15:08:43,306 | INFO  | FelixStartLevel  | core                             | 34 - org.apache.aries.jmx.core - 1.1.3 | Registering org.osgi.jmx.framework.wiring.BundleWiringStateMBean to MBeanServer com.sun.jmx.mbeanserver.JmxMBeanServer@725b1426 with name osgi.core:type=wiringState,version=1.1,framework=org.apache.felix.framework,uuid=97de0fcc-449e-4063-8c94-bd817604f89c
2015-12-08 15:08:43,306 | INFO  | FelixStartLevel  | core                             | 34 - org.apache.aries.jmx.core - 1.1.3 | Registering org.osgi.jmx.framework.BundleStateMBean to MBeanServer com.sun.jmx.mbeanserver.JmxMBeanServer@725b1426 with name osgi.core:type=bundleState,version=1.7,framework=org.apache.felix.framework,uuid=97de0fcc-449e-4063-8c94-bd817604f89c
2015-12-08 15:08:43,307 | INFO  | FelixStartLevel  | core                             | 34 - org.apache.aries.jmx.core - 1.1.3 | Registering org.osgi.jmx.service.cm.ConfigurationAdminMBean to MBeanServer com.sun.jmx.mbeanserver.JmxMBeanServer@725b1426 with name osgi.compendium:service=cm,version=1.3,framework=org.apache.felix.framework,uuid=97de0fcc-449e-4063-8c94-bd817604f89c
2015-12-08 15:08:43,836 | INFO  | FelixStartLevel  | KarArtifactInstaller             | 46 - org.apache.karaf.deployer.kar - 2.4.3 | Karaf archives will be extracted to /Users/serkanozal/Documents/tools/karaf/apache-karaf-2.4.3/system
2015-12-08 15:08:43,837 | INFO  | FelixStartLevel  | KarArtifactInstaller             | 46 - org.apache.karaf.deployer.kar - 2.4.3 | Timestamps for Karaf archives will be extracted to /Users/serkanozal/Documents/tools/karaf/apache-karaf-2.4.3/system/.timestamps
2015-12-08 15:08:43,949 | INFO  | FelixStartLevel  | ContextLoaderListener            | 70 - org.springframework.osgi.extender - 1.2.1 | Starting [org.springframework.osgi.extender] bundle v.[1.2.1]
2015-12-08 15:08:44,040 | INFO  | FelixStartLevel  | ExtenderConfiguration            | 70 - org.springframework.osgi.extender - 1.2.1 | No custom extender configuration detected; using defaults...
2015-12-08 15:08:44,045 | INFO  | FelixStartLevel  | TimerTaskExecutor                | 65 - org.apache.servicemix.bundles.spring-context - 3.2.11.RELEASE_1 | Initializing Timer
2015-12-08 15:08:44,214 | INFO  | FelixStartLevel  | Activator                        | 58 - org.apache.camel.camel-core - 2.16.0 | Camel activator starting
2015-12-08 15:08:44,224 | INFO  | FelixStartLevel  | Activator                        | 58 - org.apache.camel.camel-core - 2.16.0 | Camel activator started
2015-12-08 15:08:44,980 | INFO  | FelixStartLevel  | ultOsgiApplicationContextCreator | 70 - org.springframework.osgi.extender - 1.2.1 | Discovered configurations {osgibundle:/META-INF/spring/*.xml} in bundle [hazelcast-endpoint (com.hazelcast.api.hazelcast-endpoint)]
2015-12-08 15:08:45,008 | INFO  | ExtenderThread-1 | OsgiBundleXmlApplicationContext  | 65 - org.apache.servicemix.bundles.spring-context - 3.2.11.RELEASE_1 | Refreshing OsgiBundleXmlApplicationContext(bundle=com.hazelcast.api.hazelcast-endpoint, config=osgibundle:/META-INF/spring/*.xml): startup date [Tue Dec 08 15:08:45 EET 2015]; root of context hierarchy
2015-12-08 15:08:45,034 | INFO  | ExtenderThread-1 | OsgiBundleXmlApplicationContext  | 65 - org.apache.servicemix.bundles.spring-context - 3.2.11.RELEASE_1 | Application Context service already unpublished
2015-12-08 15:08:45,108 | INFO  | ExtenderThread-1 | XmlBeanDefinitionReader          | 63 - org.apache.servicemix.bundles.spring-beans - 3.2.11.RELEASE_1 | Loading XML bean definitions from URL [bundle://113.0:0/META-INF/spring/camel-context.xml]
2015-12-08 15:08:45,425 | INFO  | ExtenderThread-1 | XmlBeanDefinitionReader          | 63 - org.apache.servicemix.bundles.spring-beans - 3.2.11.RELEASE_1 | Loading XML bean definitions from OSGi resource[classpath:/META-INF/spring/include/beans.xml|bnd.id=113|bnd.sym=com.hazelcast.api.hazelcast-endpoint]
2015-12-08 15:08:45,539 | INFO  | ExtenderThread-1 | CamelNamespaceHandler            | 74 - org.apache.camel.camel-spring - 2.16.0 | OSGi environment detected.
2015-12-08 15:08:46,569 | INFO  | ExtenderThread-1 | WaiterApplicationContextExecutor | 70 - org.springframework.osgi.extender - 1.2.1 | No outstanding OSGi service dependencies, completing initialization for OsgiBundleXmlApplicationContext(bundle=com.hazelcast.api.hazelcast-endpoint, config=osgibundle:/META-INF/spring/*.xml)
2015-12-08 15:08:46,617 | INFO  | ExtenderThread-2 | DefaultListableBeanFactory       | 63 - org.apache.servicemix.bundles.spring-beans - 3.2.11.RELEASE_1 | Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@2c18f8fc: defining beans [org.springframework.context.annotation.internalConfigurationAnnotationProcessor,org.springframework.context.annotation.internalAutowiredAnnotationProcessor,org.springframework.context.annotation.internalRequiredAnnotationProcessor,org.springframework.context.annotation.internalCommonAnnotationProcessor,instance,map,jms,apiRouteBuilder,template,consumerTemplate,hz-api:beanPostProcessor,hz-api]; root of factory hierarchy
2015-12-08 15:08:46,670 | INFO  | ExtenderThread-2 | DefaultAddressPicker             | 113 - com.hazelcast.api.hazelcast-endpoint - 1.0.0.RC1-SNAPSHOT | [LOCAL] [dev] [3.5.3] Prefer IPv4 stack is true.
2015-12-08 15:08:46,675 | INFO  | ExtenderThread-2 | DefaultAddressPicker             | 113 - com.hazelcast.api.hazelcast-endpoint - 1.0.0.RC1-SNAPSHOT | [LOCAL] [dev] [3.5.3] Picked Address[192.168.2.110]:5701, using socket ServerSocket[addr=/0:0:0:0:0:0:0:0,localport=5701], bind any local is true
2015-12-08 15:08:46,678 | WARN  | ExtenderThread-2 | ServiceLoader                    | 113 - com.hazelcast.api.hazelcast-endpoint - 1.0.0.RC1-SNAPSHOT | Service loader could not load 'META-INF/services/com.hazelcast.instance.NodeExtension' It may be empty or does not exist.
2015-12-08 15:08:46,690 | WARN  | ExtenderThread-2 | ServiceLoader                    | 113 - com.hazelcast.api.hazelcast-endpoint - 1.0.0.RC1-SNAPSHOT | Service loader could not load 'META-INF/services/com.hazelcast.PortableHook' It may be empty or does not exist.
2015-12-08 15:08:46,693 | WARN  | ExtenderThread-2 | ServiceLoader                    | 113 - com.hazelcast.api.hazelcast-endpoint - 1.0.0.RC1-SNAPSHOT | Service loader could not load 'META-INF/services/com.hazelcast.DataSerializerHook' It may be empty or does not exist.
2015-12-08 15:08:46,704 | WARN  | ExtenderThread-2 | ServiceLoader                    | 113 - com.hazelcast.api.hazelcast-endpoint - 1.0.0.RC1-SNAPSHOT | Service loader could not load 'META-INF/services/com.hazelcast.SerializerHook' It may be empty or does not exist.
2015-12-08 15:08:46,725 | INFO  | ExtenderThread-2 | OperationService                 | 113 - com.hazelcast.api.hazelcast-endpoint - 1.0.0.RC1-SNAPSHOT | [192.168.2.110]:5701 [dev] [3.5.3] Backpressure is disabled
2015-12-08 15:08:46,737 | INFO  | ExtenderThread-2 | ClassicOperationExecutor         | 113 - com.hazelcast.api.hazelcast-endpoint - 1.0.0.RC1-SNAPSHOT | [192.168.2.110]:5701 [dev] [3.5.3] Starting with 2 generic operation threads and 4 partition operation threads.
2015-12-08 15:08:47,031 | INFO  | ExtenderThread-2 | system                           | 113 - com.hazelcast.api.hazelcast-endpoint - 1.0.0.RC1-SNAPSHOT | [192.168.2.110]:5701 [dev] [3.5.3] Hazelcast 3.5.3 (20151011 - 64c663a) starting at Address[192.168.2.110]:5701
2015-12-08 15:08:47,031 | INFO  | ExtenderThread-2 | system                           | 113 - com.hazelcast.api.hazelcast-endpoint - 1.0.0.RC1-SNAPSHOT | [192.168.2.110]:5701 [dev] [3.5.3] Copyright (c) 2008-2015, Hazelcast, Inc. All Rights Reserved.
2015-12-08 15:08:47,040 | INFO  | ExtenderThread-2 | Node                             | 113 - com.hazelcast.api.hazelcast-endpoint - 1.0.0.RC1-SNAPSHOT | [192.168.2.110]:5701 [dev] [3.5.3] Creating MulticastJoiner
2015-12-08 15:08:47,043 | INFO  | ExtenderThread-2 | LifecycleService                 | 113 - com.hazelcast.api.hazelcast-endpoint - 1.0.0.RC1-SNAPSHOT | [192.168.2.110]:5701 [dev] [3.5.3] Address[192.168.2.110]:5701 is STARTING
2015-12-08 15:08:50,178 | INFO  | ExtenderThread-2 | MulticastJoiner                  | 113 - com.hazelcast.api.hazelcast-endpoint - 1.0.0.RC1-SNAPSHOT | [192.168.2.110]:5701 [dev] [3.5.3] 


Members [1] {
    Member [192.168.2.110]:5701 this
}

2015-12-08 15:08:50,205 | INFO  | ExtenderThread-2 | LifecycleService                 | 113 - com.hazelcast.api.hazelcast-endpoint - 1.0.0.RC1-SNAPSHOT | [192.168.2.110]:5701 [dev] [3.5.3] Address[192.168.2.110]:5701 is STARTED
2015-12-08 15:08:50,610 | INFO  | ExtenderThread-2 | OsgiSpringCamelContext           | 58 - org.apache.camel.camel-core - 2.16.0 | Apache Camel 2.16.0 (CamelContext: hz-api) is starting
2015-12-08 15:08:50,610 | INFO  | ExtenderThread-2 | OsgiSpringCamelContext           | 58 - org.apache.camel.camel-core - 2.16.0 | Tracing is enabled on CamelContext: hz-api
2015-12-08 15:08:50,611 | INFO  | ExtenderThread-2 | ManagedManagementStrategy        | 58 - org.apache.camel.camel-core - 2.16.0 | JMX is enabled
2015-12-08 15:08:50,732 | WARN  | ExtenderThread-2 | DefaultTypeConverter             | 58 - org.apache.camel.camel-core - 2.16.0 | Overriding type converter from: StaticMethodTypeConverter: public static org.apache.activemq.command.ActiveMQDestination org.apache.activemq.camel.converter.ActiveMQConverter.toDestination(java.lang.String) to: StaticMethodTypeConverter: public static org.apache.activemq.command.ActiveMQDestination org.apache.activemq.camel.converter.ActiveMQConverter.toDestination(java.lang.String)
2015-12-08 15:08:50,733 | WARN  | ExtenderThread-2 | DefaultTypeConverter             | 58 - org.apache.camel.camel-core - 2.16.0 | Overriding type converter from: InstanceMethodTypeConverter: public org.apache.activemq.command.ActiveMQMessage org.apache.activemq.camel.converter.ActiveMQMessageConverter.toMessage(org.apache.camel.Exchange) throws javax.jms.JMSException to: InstanceMethodTypeConverter: public org.apache.activemq.command.ActiveMQMessage org.apache.activemq.camel.converter.ActiveMQMessageConverter.toMessage(org.apache.camel.Exchange) throws javax.jms.JMSException
2015-12-08 15:08:50,733 | WARN  | ExtenderThread-2 | DefaultTypeConverter             | 58 - org.apache.camel.camel-core - 2.16.0 | Overriding type converter from: InstanceMethodTypeConverter: public org.apache.camel.Processor org.apache.activemq.camel.converter.ActiveMQMessageConverter.toProcessor(javax.jms.MessageListener) to: InstanceMethodTypeConverter: public org.apache.camel.Processor org.apache.activemq.camel.converter.ActiveMQMessageConverter.toProcessor(javax.jms.MessageListener)
2015-12-08 15:08:50,764 | INFO  | ExtenderThread-2 | DefaultRuntimeEndpointRegistry   | 58 - org.apache.camel.camel-core - 2.16.0 | Runtime endpoint registry is in extended mode gathering usage statistics of all incoming and outgoing endpoints (cache limit: 1000)
2015-12-08 15:08:50,863 | INFO  | ExtenderThread-2 | OsgiSpringCamelContext           | 58 - org.apache.camel.camel-core - 2.16.0 | AllowUseOriginalMessage is enabled. If access to the original message is not needed, then its recommended to turn this option off as it may improve performance.
2015-12-08 15:08:50,863 | INFO  | ExtenderThread-2 | OsgiSpringCamelContext           | 58 - org.apache.camel.camel-core - 2.16.0 | StreamCaching is not in use. If using streams then its recommended to enable stream caching. See more details at http://camel.apache.org/stream-caching.html
2015-12-08 15:08:50,911 | INFO  | ExtenderThread-2 | OsgiSpringCamelContext           | 58 - org.apache.camel.camel-core - 2.16.0 | Route: route1 started and consuming from: Endpoint[direct://queue:startEndpoint]
2015-12-08 15:08:50,913 | INFO  | ExtenderThread-2 | OsgiSpringCamelContext           | 58 - org.apache.camel.camel-core - 2.16.0 | Route: route2 started and consuming from: Endpoint[direct-vm://testHZ]
2015-12-08 15:08:50,913 | INFO  | ExtenderThread-2 | OsgiSpringCamelContext           | 58 - org.apache.camel.camel-core - 2.16.0 | Total 2 routes, of which 2 is started.
2015-12-08 15:08:50,916 | INFO  | ExtenderThread-2 | OsgiSpringCamelContext           | 58 - org.apache.camel.camel-core - 2.16.0 | Apache Camel 2.16.0 (CamelContext: hz-api) started in 0.303 seconds
2015-12-08 15:08:50,920 | INFO  | ExtenderThread-2 | OsgiBundleXmlApplicationContext  | 65 - org.apache.servicemix.bundles.spring-context - 3.2.11.RELEASE_1 | Publishing application context as OSGi service with properties {org.springframework.context.service.name=com.hazelcast.api.hazelcast-endpoint, Bundle-SymbolicName=com.hazelcast.api.hazelcast-endpoint, Bundle-Version=1.0.0.RC1-SNAPSHOT}
2015-12-08 15:08:50,931 | INFO  | ExtenderThread-2 | ContextLoaderListener            | 70 - org.springframework.osgi.extender - 1.2.1 | Application context successfully refreshed (OsgiBundleXmlApplicationContext(bundle=com.hazelcast.api.hazelcast-endpoint, config=osgibundle:/META-INF/spring/*.xml))
@thomasmodeneis

This comment has been minimized.

Copy link
Author

commented Dec 8, 2015

The project will start normally in the karaf, that is expected. However the problem comes after you start using it. :(

Its really strange that you are having to add other features required, the features.xml inside the project should contain all required features, can you get me more details on this ?

I think the best way for you to be able to reproduce the same problem I'm having is by downloading/extract/using this version jboss-fuse-6.1.0.GA

I will need you you to execute the simple manual test described on the readme

If you do this with the fuse, you should be able to do this in 5 minutes, with the karaf nude and crude, you will need to configure external activemq broker.

@serkan-ozal

This comment has been minimized.

Copy link
Contributor

commented Dec 8, 2015

@thomasmodeneis

First of all, I cannot build your project without updating camel.version to 2.16.0 from 2.12.0.redhat-611412.

Then I have downloaded JBoss-Fuse from the link you provided and started Karaf via karaf under bin directory.
And added URL for Hazelcast-Endpoint by addurl mvn:com.hazelcast.api/hazelcast-endpoint/1.0.0.RC1-SNAPSHOT/xml/features as you mentioned.

So far so good, but when I have try to install the Hazelcast-Endpoint, Karaf gives me this error as I had faced before:

Error executing command: Could not start bundle mvn:com.hazelcast.api/hazelcast-endpoint/1.0.0.RC1-SNAPSHOT in feature(s) hazelcast-endpoint-1.0.0.RC1-SNAPSHOT: Unresolved constraint in bundle com.hazelcast.api.hazelcast-endpoint [258]: Unable to resolve 258.0: missing requirement [258.0] osgi.wiring.package; (&(osgi.wiring.package=org.apache.camel)(version>=2.16.0)(!(version>=3.0.0)))

This is why I had added other dependencies.

@thomasmodeneis

This comment has been minimized.

Copy link
Author

commented Dec 8, 2015

Right, this error is because you updated the camel version, you should not do this, otherwise the problem will become even bigger (this is OSGi)

If you configure your mvn to search for jars on jboss repo, this dependency should be managed without problems, or you can download and install manually:

https://repo.fusesource.com/nexus/content/groups/public/

Full path
https://repo.fusesource.com/nexus/content/groups/public/org/apache/camel/camel-core/2.12.0.redhat-611412/

@serkan-ozal

This comment has been minimized.

Copy link
Contributor

commented Dec 8, 2015

@thomasmodeneis

It would be better, if you had added the repository to pom.xml like this:

    <repositories>
        <repository>
            <id>fuse</id>
            <url>https://repo.fusesource.com/nexus/content/groups/public/</url>
        </repository>
    </repositories>

Anyway, I have started and run your sample on JBoss-Fuse.
Then I have sent the test JSON context to startEndpoint queue and see this result without any exception.

{"coord":{"lon":4.89,"lat":52.37},"base":"testhz"}
done
@thomasmodeneis

This comment has been minimized.

Copy link
Author

commented Dec 8, 2015

Sorry about the repositories thing, this is because at the client they have their own nexus repo and I completely forgot to mention it to you, I've updated the git project with the repositories thing.

Great, it works and it was supposed to work.

Can you check on the logs to see if you see the original warn message that was the origin of this ticket ?

2015-11-23 10:10:36,448 | WARN  | .MulticastThread | MulticastService                 | ?                                   ? | 274 - com.hazelcast-endpoint - 1.0.0.RC1-SNAPSHOT | [10.0.2.15]:5701 [dev] [3.5.3] Received data format is invalid. (An old version of Hazelcast may be running here.)
@serkan-ozal

This comment has been minimized.

Copy link
Contributor

commented Dec 8, 2015

I have checked but didn't see an exception and warn message like above

@serkan-ozal

This comment has been minimized.

Copy link
Contributor

commented Dec 8, 2015

Are you able reproduce this issue with the sample you provided

@thomasmodeneis

This comment has been minimized.

Copy link
Author

commented Dec 8, 2015

I'm using Fuse 6.1.1 and it does happen the warn message saying there is another HZ running already (weird).
Can you just confirm to me if you completed the test with Fuse 6.1.GA or you did it with Karaf 2.4.3 ?

Thanks

@serkan-ozal

This comment has been minimized.

Copy link
Contributor

commented Dec 8, 2015

Yes, I have used jboss-fuse-6.1.0.GA you mentioned in the test

@thomasmodeneis

This comment has been minimized.

Copy link
Author

commented Dec 8, 2015

Okay, well I guess this is an issue with the current version of the Fuse my client is using.
I will just do a quick test with jboss-fuse-6.1.0.GA and if all goes well I will close it here, thank you for your great support mate, that was quite helpful :)
Thanks

@serkan-ozal

This comment has been minimized.

Copy link
Contributor

commented Dec 8, 2015

can you share your full logs?

@serkan-ozal

This comment has been minimized.

Copy link
Contributor

commented Dec 8, 2015

In addition, there are logs like this

Service loader could not load 'META-INF/services/com.hazelcast.XYZ' It may be empty or does not exist.

and they are suspicious for me

@thomasmodeneis

This comment has been minimized.

Copy link
Author

commented Dec 8, 2015

When I started to develop this project, I struggled ALLOT to find out the right mvn configurations and I also messed with several internal karaf configurations in order to have HZ working with Karaf and I'm starting to think that somehow that affected and produced the behavior described on this ticket.

I've completely removed the old jboss fuse instalation and replaced with one fresh using the very same version, and I got the demo working and no warns thrown except for this very same you also reported.

I've added the logs to the repo:
https://github.com/thomasmodeneis/hazelcast-fuse61/blob/master/src/test/logs/test.log

Thanks

@serkan-ozal

This comment has been minimized.

Copy link
Contributor

commented Dec 8, 2015

@thomasmodeneis

I have noticed a problem in the generated jar hazelcast-endpoint-1.0.0.RC1-SNAPSHOT.jar

There are embedded Hazelcast classes but not services folder META-INF/services/.

There might be a problem in your maven plugin configuration.

@thomasmodeneis

This comment has been minimized.

Copy link
Author

commented Dec 9, 2015

Right I concluded the same here, the main problem is how to configure maven to extract this folder from inside the hazelcast
Do you have any ideas ?

@serkan-ozal

This comment has been minimized.

Copy link
Contributor

commented Dec 9, 2015

@thomasmodeneis

Can you share your pom.xml with your changes?

Also from the stacktrace, I see that the root issue comes from Felix due to the bundle is uninstalled.
I think, in fact this is not Hazelcast (serialization issue)

@thomasmodeneis

This comment has been minimized.

Copy link
Author

commented Dec 9, 2015

I've pushed in the changes to the github project, you can just pull it from there.

I seems from what I can understand, the bundle is un-installed but somehow felix is not able to completely remove it from the container, and the hazelcast bundle stays alive there forever causing this issue. It seems like a orphan bundle just consuming resources.

@serkan-ozal

This comment has been minimized.

Copy link
Contributor

commented Dec 9, 2015

Ok, I have pulled the code.
By the way, without uninstall, two Hazelcast instance can see and communicate with each other on JBoss Fuse without serialization issue so this make me think that issue is related with outside of Hazelcast.

@thomasmodeneis

This comment has been minimized.

Copy link
Author

commented Dec 9, 2015

I've never tested to have two instances communicating on the same container so I can't make a point on this topic.
It must be something related to the Karaf+Felix integration with HZ.
Do you have a working example with HZ working with a OSGi containner that we can compare, maybe ?

@serkan-ozal

This comment has been minimized.

@thomasmodeneis

This comment has been minimized.

Copy link
Author

commented Dec 9, 2015

I see you have implemented a very basic demo. Let me reformulate the question, do we have a real world example with real usage of the cache (put/get/del) operations on a production ready container (karaf) ?

@serkan-ozal

This comment has been minimized.

Copy link
Contributor

commented Dec 9, 2015

unfortunately not

@thomasmodeneis

This comment has been minimized.

Copy link
Author

commented Dec 9, 2015

Great, so actually this ticket is a very good place for us to find out what is wrong with the HZ component and make use of it in the official HZ examples folder (when this happens to work properly).
Do you know anyone else inside the HZ that can give us some insights of what to look for in order to get this cleared ?

@serkan-ozal

This comment has been minimized.

Copy link
Contributor

commented Dec 9, 2015

can you try it also with latest Hazelcast version also (It should be 3.6-EA2)

@thomasmodeneis

This comment has been minimized.

Copy link
Author

commented Dec 9, 2015

Yes, I will be doing this asap. Right now I got into bigger problem so It may take a day for me to get free again. I will keep you posted.

@mesutcelik mesutcelik modified the milestones: 3.6, 3.6.1 Dec 10, 2015

@serkan-ozal serkan-ozal modified the milestones: 3.6.2, 3.6.1 Feb 23, 2016

@serkan-ozal

This comment has been minimized.

Copy link
Contributor

commented Mar 11, 2016

Hi @thomasmodeneis,
Any update on this issue?

@serkan-ozal

This comment has been minimized.

Copy link
Contributor

commented Mar 22, 2016

Hi @thomasmodeneis closing this issue now and please feel free to reopen the issue if the problem still occurs

@thomasmodeneis

This comment has been minimized.

Copy link
Author

commented Mar 22, 2016

Hi @serkan-ozal
I've updated the project pom to use the recommended 3.6-EA2, however this seem to have changed something and my test class wont startup and throws exception,

I've raised a ticket on my repo soujava/hazelcast-fuse61#6 with more details on how to reproduce the issue, can you please have a look if you can find what I'm missing to update on my code as part of this upgrade ?
Thanks

RemoteTestNG starting
org.testng.TestNGException: 
Cannot instantiate class com.hazelcast.cache.CacheUnitTest
    at org.testng.internal.ObjectFactoryImpl.newInstance(ObjectFactoryImpl.java:37)
    at org.testng.internal.ClassHelper.createInstance1(ClassHelper.java:372)
    at org.testng.internal.ClassHelper.createInstance(ClassHelper.java:284)
    at org.testng.internal.ClassImpl.getDefaultInstance(ClassImpl.java:109)
    at org.testng.internal.ClassImpl.getInstances(ClassImpl.java:181)
    at org.testng.internal.TestNGClassFinder.<init>(TestNGClassFinder.java:120)
    at org.testng.TestRunner.initMethods(TestRunner.java:402)
    at org.testng.TestRunner.init(TestRunner.java:228)
    at org.testng.TestRunner.init(TestRunner.java:198)
    at org.testng.TestRunner.<init>(TestRunner.java:154)
    at org.testng.remote.RemoteTestNG$1.newTestRunner(RemoteTestNG.java:141)
    at org.testng.remote.RemoteTestNG$DelegatingTestRunnerFactory.newTestRunner(RemoteTestNG.java:270)
    at org.testng.SuiteRunner$ProxyTestRunnerFactory.newTestRunner(SuiteRunner.java:561)
    at org.testng.SuiteRunner.init(SuiteRunner.java:156)
    at org.testng.SuiteRunner.<init>(SuiteRunner.java:110)
    at org.testng.TestNG.createSuiteRunner(TestNG.java:1235)
    at org.testng.TestNG.createSuiteRunners(TestNG.java:1222)
    at org.testng.TestNG.runSuitesLocally(TestNG.java:1074)
    at org.testng.TestNG.run(TestNG.java:999)
    at org.testng.remote.RemoteTestNG.run(RemoteTestNG.java:111)
    at org.testng.remote.RemoteTestNG.initAndRun(RemoteTestNG.java:203)
    at org.testng.remote.RemoteTestNG.main(RemoteTestNG.java:174)
    at org.testng.RemoteTestNGStarter.main(RemoteTestNGStarter.java:125)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:140)
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
    at org.testng.internal.ObjectFactoryImpl.newInstance(ObjectFactoryImpl.java:28)
    ... 27 more
Caused by: java.lang.NumberFormatException: null
    at java.lang.Integer.parseInt(Integer.java:454)
    at java.lang.Byte.parseByte(Byte.java:148)
    at java.lang.Byte.parseByte(Byte.java:174)
    at com.hazelcast.instance.BuildInfoProvider.getBuildInfo(BuildInfoProvider.java:67)
    at com.hazelcast.config.AbstractXmlConfigHelper.getReleaseVersion(AbstractXmlConfigHelper.java:210)
    at com.hazelcast.config.AbstractXmlConfigHelper.<init>(AbstractXmlConfigHelper.java:72)
    at com.hazelcast.config.AbstractConfigBuilder.<init>(AbstractConfigBuilder.java:60)
    at com.hazelcast.config.XmlConfigBuilder.<init>(XmlConfigBuilder.java:156)
    at com.hazelcast.instance.HazelcastInstanceFactory.newHazelcastInstance(HazelcastInstanceFactory.java:109)
    at com.hazelcast.core.Hazelcast.newHazelcastInstance(Hazelcast.java:58)
    at com.hazelcast.cache.CacheUnitTest.<init>(CacheUnitTest.java:17)
    ... 32 more
RemoteTestNG finishing: -1458640147813 ms

Process finished with exit code 0
@serkan-ozal

This comment has been minimized.

Copy link
Contributor

commented Mar 22, 2016

Hi @thomasmodeneis,

I think, hazelcast-runtime.properties is not there now like META-INF/services/ before. Can you include hazelcast-runtime.properties by the same way you did for META-INF/services/.

I mean. something like this:

<Private-Package>
    hazelcast-runtime.properties,
    ...
</Private-Package>

Then please be sure that hazelcast-runtime.properties is there

@thomasmodeneis

This comment has been minimized.

Copy link
Author

commented Mar 23, 2016

Hi @serkan-ozal
Thanks for your tip, however I'm still stuck in the same error, nothing changed.

I see the error happens here:

        String sv = runtimeProperties.getProperty("hazelcast.serialization.version");
        byte serialVersion = Byte.parseByte(sv);

When looking into the hz-runtime, I see I don't have anything like this there, do you know what else maybe wrong ?

hazelcast.version = 3.5.3
hazelcast.build = 20151011
hazelcast.distribution = Hazelcast
hazelcast.git.revision = 64c663a
@serkan-ozal

This comment has been minimized.

Copy link
Contributor

commented Mar 23, 2016

@thomasmodeneis
Don't you use 3.6-EA2 or later version?
Because somehow, your hazelcast-runtime.properties belongs to older version.

@thomasmodeneis

This comment has been minimized.

Copy link
Author

commented Mar 24, 2016

Hi @serkan-ozal
I see that the hz.runtime.prop is old, I've been digging into the release file to get the new version however right now I can't use this without having the real vars:

hazelcast.version = ${project.version}
hazelcast.build = ${timestamp}
hazelcast.distribution = ${hazelcast.distribution}
hazelcast.git.revision = ${git.commit.id.abbrev}
hazelcast.serialization.version = ${hazelcast.serialization.version}

Can you please provide me the correct values for this ?
Thanks

@serkan-ozal

This comment has been minimized.

Copy link
Contributor

commented Mar 24, 2016

Hi @thomasmodeneis
You can find these values from the jar you are using.
For 3.6-EA2 they are:

hazelcast.version = 3.6-EA2
hazelcast.build = 20151125
hazelcast.distribution = Hazelcast
hazelcast.git.revision = 8af85c5
hazelcast.serialization.version = 1
@thomasmodeneis

This comment has been minimized.

Copy link
Author

commented Mar 25, 2016

I've solved this one and couple others but when starting now there is a new dependency :(

com.hazelcast.core.Hazelcast.newHazelcastInstance(com.hazelcast.config.Config)] threw exception; nested exception is java.lang.NoClassDefFoundError: javax/security/auth/callback/UnsupportedCallbackException

I will go on trying to overcome this and possible more OSGi related issues today and will keep you posted on the progress, cheers.

@thomasmodeneis

This comment has been minimized.

Copy link
Author

commented Mar 25, 2016

I've noticed that with this new version, when I remove the bundle from the container a new exception is thrown:

JBossFuse:karaf@root> features:Exception in thread "hz._hzInstance_1_dev.migration" java.lang.NoClassDefFoundError: com/hazelcast/partition/impl/InternalPartitionServiceImpl$MigrateTask
    at com.hazelcast.partition.impl.InternalPartitionServiceImpl$MigrationThread.processTask(InternalPartitionServiceImpl.java:2119)
    at com.hazelcast.partition.impl.InternalPartitionServiceImpl$MigrationThread.doRun(InternalPartitionServiceImpl.java:2097)
    at com.hazelcast.partition.impl.InternalPartitionServiceImpl$MigrationThread.run(InternalPartitionServiceImpl.java:2074)
Caused by: java.lang.ClassNotFoundException: Unable to load class 'com.hazelcast.partition.impl.InternalPartitionServiceImpl$MigrateTask' because the bundle wiring for com.hazelcast.api.hazelcast-endpoint is no longer valid.
    at org.apache.felix.framework.BundleWiringImpl.findClassOrResourceByDelegation(BundleWiringImpl.java:1494)
    at org.apache.felix.framework.BundleWiringImpl.access$400(BundleWiringImpl.java:75)
    at org.apache.felix.framework.BundleWiringImpl$BundleClassLoader.loadClass(BundleWiringImpl.java:1955)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
    ... 3 more
@thomasmodeneis

This comment has been minimized.

Copy link
Author

commented Mar 25, 2016

I've managed to overcome all the issues, however now HZ is throwing this exception all the time filling up the logs with it:

listException in thread "cached7" java.lang.NoClassDefFoundError: com/hazelcast/cluster/impl/MulticastJoiner$1
    at com.hazelcast.cluster.impl.MulticastJoiner.searchForOtherClusters(MulticastJoiner.java:106)
    at com.hazelcast.cluster.impl.SplitBrainHandler.searchForOtherClusters(SplitBrainHandler.java:72)
    at com.hazelcast.cluster.impl.SplitBrainHandler.run(SplitBrainHandler.java:41)
    at com.hazelcast.util.executor.CachedExecutorServiceDelegate$Worker.run(CachedExecutorServiceDelegate.java:212)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
    at com.hazelcast.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76)
    at com.hazelcast.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:92)
@thomasmodeneis

This comment has been minimized.

Copy link
Author

commented Mar 25, 2016

And the reported problem still there:

2016-03-25 10:30:37,008 | WARN  | .MulticastThread | MulticastService                 | ?                                   ? | 343 - com.hazelcast.api.hazelcast-endpoint - 1.0.0.RC2-SNAPSHOT | [172.17.0.1]:5701 [dev] [3.6-EA2] Received data format is invalid. (An old version of Hazelcast may be running here.)
com.hazelcast.nio.serialization.HazelcastSerializationException: java.lang.IllegalStateException: Bundle is uninstalled
    at com.hazelcast.internal.serialization.impl.SerializationUtil.handleException(SerializationUtil.java:73)[343:com.hazelcast.api.hazelcast-endpoint:1.0.0.RC2-SNAPSHOT]
    at com.hazelcast.internal.serialization.impl.AbstractSerializationService.readObject(AbstractSerializationService.java:211)[343:com.hazelcast.api.hazelcast-endpoint:1.0.0.RC2-SNAPSHOT]
    at com.hazelcast.internal.serialization.impl.ByteArrayObjectDataInput.readObject(ByteArrayObjectDataInput.java:600)[343:com.hazelcast.api.hazelcast-endpoint:1.0.0.RC2-SNAPSHOT]
    at com.hazelcast.cluster.impl.ConfigCheck.readData(ConfigCheck.java:215)[343:com.hazelcast.api.hazelcast-endpoint:1.0.0.RC2-SNAPSHOT]
    at com.hazelcast.cluster.impl.JoinMessage.readData(JoinMessage.java:98)[343:com.hazelcast.api.hazelcast-endpoint:1.0.0.RC2-SNAPSHOT]
    at com.hazelcast.cluster.impl.JoinRequest.readData(JoinRequest.java:68)[343:com.hazelcast.api.hazelcast-endpoint:1.0.0.RC2-SNAPSHOT]
    at com.hazelcast.internal.serialization.impl.DataSerializer.read(DataSerializer.java:121)[343:com.hazelcast.api.hazelcast-endpoint:1.0.0.RC2-SNAPSHOT]
    at com.hazelcast.internal.serialization.impl.DataSerializer.read(DataSerializer.java:47)[343:com.hazelcast.api.hazelcast-endpoint:1.0.0.RC2-SNAPSHOT]
    at com.hazelcast.internal.serialization.impl.StreamSerializerAdapter.read(StreamSerializerAdapter.java:46)[343:com.hazelcast.api.hazelcast-endpoint:1.0.0.RC2-SNAPSHOT]
    at com.hazelcast.internal.serialization.impl.AbstractSerializationService.readObject(AbstractSerializationService.java:205)[343:com.hazelcast.api.hazelcast-endpoint:1.0.0.RC2-SNAPSHOT]
    at com.hazelcast.internal.serialization.impl.ByteArrayObjectDataInput.readObject(ByteArrayObjectDataInput.java:600)[343:com.hazelcast.api.hazelcast-endpoint:1.0.0.RC2-SNAPSHOT]
    at com.hazelcast.cluster.impl.MulticastService.receive(MulticastService.java:201)[343:com.hazelcast.api.hazelcast-endpoint:1.0.0.RC2-SNAPSHOT]
    at com.hazelcast.cluster.impl.MulticastService.run(MulticastService.java:159)[343:com.hazelcast.api.hazelcast-endpoint:1.0.0.RC2-SNAPSHOT]
    at java.lang.Thread.run(Thread.java:745)[:1.7.0_91]
Caused by: java.lang.IllegalStateException: Bundle is uninstalled
    at org.apache.felix.framework.Felix.loadBundleClass(Felix.java:1840)[org.apache.felix.framework-4.0.3.redhat-611412.jar:]
    at org.apache.felix.framework.BundleImpl.loadClass(BundleImpl.java:937)[org.apache.felix.framework-4.0.3.redhat-611412.jar:]
    at org.springframework.osgi.util.BundleDelegatingClassLoader.findClass(BundleDelegatingClassLoader.java:99)[147:org.springframework.osgi.core:1.2.1]
    at org.springframework.osgi.util.BundleDelegatingClassLoader.loadClass(BundleDelegatingClassLoader.java:156)[147:org.springframework.osgi.core:1.2.1]
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)[:1.7.0_91]
    at com.hazelcast.nio.ClassLoaderUtil.tryLoadClass(ClassLoaderUtil.java:137)[343:com.hazelcast.api.hazelcast-endpoint:1.0.0.RC2-SNAPSHOT]
    at com.hazelcast.nio.ClassLoaderUtil.loadClass(ClassLoaderUtil.java:99)[343:com.hazelcast.api.hazelcast-endpoint:1.0.0.RC2-SNAPSHOT]
    at com.hazelcast.nio.IOUtil$1.resolveClass(IOUtil.java:115)[343:com.hazelcast.api.hazelcast-endpoint:1.0.0.RC2-SNAPSHOT]
    at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1612)[:1.7.0_91]
    at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517)[:1.7.0_91]
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)[:1.7.0_91]
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)[:1.7.0_91]
    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)[:1.7.0_91]
    at com.hazelcast.internal.serialization.impl.JavaDefaultSerializers$JavaSerializer.read(JavaDefaultSerializers.java:219)[343:com.hazelcast.api.hazelcast-endpoint:1.0.0.RC2-SNAPSHOT]
    at com.hazelcast.internal.serialization.impl.StreamSerializerAdapter.read(StreamSerializerAdapter.java:46)[343:com.hazelcast.api.hazelcast-endpoint:1.0.0.RC2-SNAPSHOT]
    at com.hazelcast.internal.serialization.impl.AbstractSerializationService.readObject(AbstractSerializationService.java:205)[343:com.hazelcast.api.hazelcast-endpoint:1.0.0.RC2-SNAPSHOT]
    ... 12 more
2016-03-25 10:30:37,007 | WARN  | .MulticastThread | MulticastService                 | ?                                   ? | 344 - com.hazelcast.api.hazelcast-endpoint - 1.0.0.RC2-SNAPSHOT | [172.17.0.1]:5702 [dev] [3.6-EA2] Received data format is invalid. (An old version of Hazelcast may be running here.)
com.hazelcast.nio.serialization.HazelcastSerializationException: java.lang.IllegalStateException: Bundle is uninstalled
    at com.hazelcast.internal.serialization.impl.SerializationUtil.handleException(SerializationUtil.java:73)[344:com.hazelcast.api.hazelcast-endpoint:1.0.0.RC2-SNAPSHOT]
    at com.hazelcast.internal.serialization.impl.AbstractSerializationService.readObject(AbstractSerializationService.java:211)[344:com.hazelcast.api.hazelcast-endpoint:1.0.0.RC2-SNAPSHOT]
    at com.hazelcast.internal.serialization.impl.ByteArrayObjectDataInput.readObject(ByteArrayObjectDataInput.java:600)[344:com.hazelcast.api.hazelcast-endpoint:1.0.0.RC2-SNAPSHOT]
    at com.hazelcast.cluster.impl.ConfigCheck.readData(ConfigCheck.java:215)[344:com.hazelcast.api.hazelcast-endpoint:1.0.0.RC2-SNAPSHOT]
    at com.hazelcast.cluster.impl.JoinMessage.readData(JoinMessage.java:98)[344:com.hazelcast.api.hazelcast-endpoint:1.0.0.RC2-SNAPSHOT]
    at com.hazelcast.cluster.impl.JoinRequest.readData(JoinRequest.java:68)[344:com.hazelcast.api.hazelcast-endpoint:1.0.0.RC2-SNAPSHOT]
    at com.hazelcast.internal.serialization.impl.DataSerializer.read(DataSerializer.java:121)[344:com.hazelcast.api.hazelcast-endpoint:1.0.0.RC2-SNAPSHOT]
    at com.hazelcast.internal.serialization.impl.DataSerializer.read(DataSerializer.java:47)[344:com.hazelcast.api.hazelcast-endpoint:1.0.0.RC2-SNAPSHOT]
    at com.hazelcast.internal.serialization.impl.StreamSerializerAdapter.read(StreamSerializerAdapter.java:46)[344:com.hazelcast.api.hazelcast-endpoint:1.0.0.RC2-SNAPSHOT]
    at com.hazelcast.internal.serialization.impl.AbstractSerializationService.readObject(AbstractSerializationService.java:205)[344:com.hazelcast.api.hazelcast-endpoint:1.0.0.RC2-SNAPSHOT]
    at com.hazelcast.internal.serialization.impl.ByteArrayObjectDataInput.readObject(ByteArrayObjectDataInput.java:600)[344:com.hazelcast.api.hazelcast-endpoint:1.0.0.RC2-SNAPSHOT]
    at com.hazelcast.cluster.impl.MulticastService.receive(MulticastService.java:201)[344:com.hazelcast.api.hazelcast-endpoint:1.0.0.RC2-SNAPSHOT]
    at com.hazelcast.cluster.impl.MulticastService.run(MulticastService.java:159)[344:com.hazelcast.api.hazelcast-endpoint:1.0.0.RC2-SNAPSHOT]
    at java.lang.Thread.run(Thread.java:745)[:1.7.0_91]
Caused by: java.lang.IllegalStateException: Bundle is uninstalled
    at org.apache.felix.framework.Felix.loadBundleClass(Felix.java:1840)[org.apache.felix.framework-4.0.3.redhat-611412.jar:]
    at org.apache.felix.framework.BundleImpl.loadClass(BundleImpl.java:937)[org.apache.felix.framework-4.0.3.redhat-611412.jar:]
    at org.springframework.osgi.util.BundleDelegatingClassLoader.findClass(BundleDelegatingClassLoader.java:99)[147:org.springframework.osgi.core:1.2.1]
    at org.springframework.osgi.util.BundleDelegatingClassLoader.loadClass(BundleDelegatingClassLoader.java:156)[147:org.springframework.osgi.core:1.2.1]
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)[:1.7.0_91]
    at com.hazelcast.nio.ClassLoaderUtil.tryLoadClass(ClassLoaderUtil.java:137)[344:com.hazelcast.api.hazelcast-endpoint:1.0.0.RC2-SNAPSHOT]
    at com.hazelcast.nio.ClassLoaderUtil.loadClass(ClassLoaderUtil.java:99)[344:com.hazelcast.api.hazelcast-endpoint:1.0.0.RC2-SNAPSHOT]
    at com.hazelcast.nio.IOUtil$1.resolveClass(IOUtil.java:115)[344:com.hazelcast.api.hazelcast-endpoint:1.0.0.RC2-SNAPSHOT]
    at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1612)[:1.7.0_91]
    at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517)[:1.7.0_91]
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)[:1.7.0_91]
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)[:1.7.0_91]
    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)[:1.7.0_91]
    at com.hazelcast.internal.serialization.impl.JavaDefaultSerializers$JavaSerializer.read(JavaDefaultSerializers.java:219)[344:com.hazelcast.api.hazelcast-endpoint:1.0.0.RC2-SNAPSHOT]
    at com.hazelcast.internal.serialization.impl.StreamSerializerAdapter.read(StreamSerializerAdapter.java:46)[344:com.hazelcast.api.hazelcast-endpoint:1.0.0.RC2-SNAPSHOT]
    at com.hazelcast.internal.serialization.impl.AbstractSerializationService.readObject(AbstractSerializationService.java:205)[344:com.hazelcast.api.hazelcast-endpoint:1.0.0.RC2-SNAPSHOT]
    ... 12 more
@thomasmodeneis

This comment has been minimized.

Copy link
Author

commented Mar 25, 2016

I've updated the repo https://github.com/thomasmodeneis/hazelcast-fuse61 with the latest code, its using the recommended version and evidences that the proposed solution did not work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.