Can not start ES after upgrade to 1.5.0

[cyberhuman]

We had to revert to 1.4.4 because 1.5.0 won't start. What might be causing it?

[2015-03-26 08:40:36,558][INFO ][node                     ] [lepton001] version[1.5.0], pid[13791], build[5448160/2015-03-23T14:30:58Z]
[2015-03-26 08:40:36,558][INFO ][node                     ] [lepton001] initializing ...
[2015-03-26 08:40:36,624][ERROR][plugins                  ] [lepton001] cannot start plugin due to incorrect Lucene version: plugin [4.10.2], node [4.10.4].
[2015-03-26 08:40:36,625][WARN ][plugins                  ] [lepton001] failed to load plugin from [jar:file:/usr/share/elasticsearch/plugins/experimental-highlighter-elasticsearch-plugin/experimental-highlighter-elasticsearch-plugin-1.4.1
.jar!/es-plugin.properties]
org.elasticsearch.ElasticsearchException: Failed to load plugin class [org.wikimedia.highlighter.experimental.elasticsearch.plugin.ExperimentalHighlighterPlugin]
        at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:533)
        at org.elasticsearch.plugins.PluginsService.loadPluginsFromClasspath(PluginsService.java:408)
        at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:116)
        at org.elasticsearch.node.internal.InternalNode.<init>(InternalNode.java:152)
        at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:159)
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:70)
        at org.elasticsearch.bootstrap.Bootstrap.main(Bootstrap.java:213)
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:32)
Caused by: org.elasticsearch.ElasticsearchException: Plugin is incompatible with the current node
        at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:526)
        ... 7 more
[2015-03-26 08:40:36,629][INFO ][plugins                  ] [lepton001] loaded [analysis-smartcn, analysis-kuromoji, marvel, analysis-combo, analysis-icu], sites [marvel, head, kopf, paramedic, HQ]
[2015-03-26 08:40:37,451][ERROR][bootstrap                ] Exception
org.elasticsearch.common.util.concurrent.ExecutionError: org.elasticsearch.common.util.concurrent.ExecutionError: java.lang.NoClassDefFoundError: Lorg/elasticsearch/indices/InternalIndicesService;
        at org.elasticsearch.common.cache.LocalCache$Segment.get(LocalCache.java:2201)
        at org.elasticsearch.common.cache.LocalCache.get(LocalCache.java:3937)
        at org.elasticsearch.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
        at org.elasticsearch.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
        at org.elasticsearch.common.inject.internal.FailableCache.get(FailableCache.java:51)
        at org.elasticsearch.common.inject.ConstructorInjectorStore.get(ConstructorInjectorStore.java:50)
        at org.elasticsearch.common.inject.ConstructorBindingImpl.initialize(ConstructorBindingImpl.java:50)
        at org.elasticsearch.common.inject.InjectorImpl.initializeBinding(InjectorImpl.java:372)
        at org.elasticsearch.common.inject.BindingProcessor$1$1.run(BindingProcessor.java:148)
        at org.elasticsearch.common.inject.BindingProcessor.initializeBindings(BindingProcessor.java:204)
        at org.elasticsearch.common.inject.InjectorBuilder.initializeStatically(InjectorBuilder.java:119)
        at org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:102)
        at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:93)
        at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:70)
        at org.elasticsearch.common.inject.ModulesBuilder.createInjector(ModulesBuilder.java:59)
        at org.elasticsearch.node.internal.InternalNode.<init>(InternalNode.java:203)
        at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:159)
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:70)
        at org.elasticsearch.bootstrap.Bootstrap.main(Bootstrap.java:213)
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:32)
Caused by: org.elasticsearch.common.util.concurrent.ExecutionError: java.lang.NoClassDefFoundError: Lorg/elasticsearch/indices/InternalIndicesService;
        at org.elasticsearch.common.cache.LocalCache$Segment.get(LocalCache.java:2201)
        at org.elasticsearch.common.cache.LocalCache.get(LocalCache.java:3937)
        at org.elasticsearch.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
        at org.elasticsearch.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
        at org.elasticsearch.common.inject.internal.FailableCache.get(FailableCache.java:51)
        at org.elasticsearch.common.inject.MembersInjectorStore.get(MembersInjectorStore.java:68)
        at org.elasticsearch.common.inject.ConstructorInjectorStore.createConstructor(ConstructorInjectorStore.java:67)
        at org.elasticsearch.common.inject.ConstructorInjectorStore.access$000(ConstructorInjectorStore.java:29)
        at org.elasticsearch.common.inject.ConstructorInjectorStore$1.create(ConstructorInjectorStore.java:37)
        at org.elasticsearch.common.inject.ConstructorInjectorStore$1.create(ConstructorInjectorStore.java:33)
        at org.elasticsearch.common.inject.internal.FailableCache$1.load(FailableCache.java:39)
        at org.elasticsearch.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
        at org.elasticsearch.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
        at org.elasticsearch.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
        at org.elasticsearch.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
        ... 19 more
Caused by: java.lang.NoClassDefFoundError: Lorg/elasticsearch/indices/InternalIndicesService;
        at java.lang.Class.getDeclaredFields0(Native Method)
        at java.lang.Class.privateGetDeclaredFields(Class.java:2499)
        at java.lang.Class.getDeclaredFields(Class.java:1811)
        at org.elasticsearch.common.inject.spi.InjectionPoint$Factory$1.getMembers(InjectionPoint.java:378)
        at org.elasticsearch.common.inject.spi.InjectionPoint$Factory$1.getMembers(InjectionPoint.java:376)
        at org.elasticsearch.common.inject.spi.InjectionPoint.addInjectorsForMembers(InjectionPoint.java:351)
        at org.elasticsearch.common.inject.spi.InjectionPoint.addInjectionPoints(InjectionPoint.java:345)
        at org.elasticsearch.common.inject.spi.InjectionPoint.forInstanceMethodsAndFields(InjectionPoint.java:287)
        at org.elasticsearch.common.inject.MembersInjectorStore.createWithListeners(MembersInjectorStore.java:80)
        at org.elasticsearch.common.inject.MembersInjectorStore.access$000(MembersInjectorStore.java:36)
        at org.elasticsearch.common.inject.MembersInjectorStore$1.create(MembersInjectorStore.java:45)
        at org.elasticsearch.common.inject.MembersInjectorStore$1.create(MembersInjectorStore.java:41)
        at org.elasticsearch.common.inject.internal.FailableCache$1.load(FailableCache.java:39)
        at org.elasticsearch.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
        at org.elasticsearch.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
        at org.elasticsearch.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
        at org.elasticsearch.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
        ... 33 more
Caused by: java.lang.ClassNotFoundException: org.elasticsearch.indices.InternalIndicesService
        at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
        ... 50 more

ES is installed from the debian repo:

$ apt-cache policy elasticsearch
elasticsearch:
  Installed: 1.5.0
  Candidate: 1.5.0
  Version table:
 *** 1.5.0 0
        990 http://packages.elasticsearch.org/elasticsearch/1.5/debian/ stable/main amd64 Packages
        100 /var/lib/dpkg/status
     1.0.3+dfsg-5 0
        500 http://debian.mirrors.ovh.net/debian/ jessie/main amd64 Packages

We are using Java7 from Oracle:

java version "1.7.0_76"
Java(TM) SE Runtime Environment (build 1.7.0_76-b13)
Java HotSpot(TM) 64-Bit Server VM (build 24.76-b04, mixed mode)

Let me know if more information is needed.
Thanks.

[tlrx]

It looks like this plugin is not compatible with ES 1.5. You have to use a version of the plugin which is compatible with elasticsearch 1.5.

[cyberhuman]

Hi Tanguy,
I removed ExperimentalHighlighterPlugin, that was not compatible, but the main issue (ClassNotFoundException: org.elasticsearch.indices.InternalIndicesService) still remains.

[tlrx]

Can you please post again the output without the ExperimentalHighlighterPlugin?

[cyberhuman]

Here you are:

[2015-03-26 09:38:58,645][INFO ][node                     ] [lepton001] version[1.5.0], pid[5926], build[5448160/2015-03-23T14:30:58Z]
[2015-03-26 09:38:58,646][INFO ][node                     ] [lepton001] initializing ...
[2015-03-26 09:38:58,706][INFO ][plugins                  ] [lepton001] loaded [analysis-smartcn, analysis-kuromoji, marvel, analysis-combo, analysis-icu], sites [marvel, head, kopf, paramedic, HQ]
[2015-03-26 09:38:59,532][ERROR][bootstrap                ] Exception
org.elasticsearch.common.util.concurrent.ExecutionError: org.elasticsearch.common.util.concurrent.ExecutionError: java.lang.NoClassDefFoundError: Lorg/elasticsearch/indices/InternalIndicesService;
        at org.elasticsearch.common.cache.LocalCache$Segment.get(LocalCache.java:2201)
        at org.elasticsearch.common.cache.LocalCache.get(LocalCache.java:3937)
        at org.elasticsearch.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
        at org.elasticsearch.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
        at org.elasticsearch.common.inject.internal.FailableCache.get(FailableCache.java:51)
        at org.elasticsearch.common.inject.ConstructorInjectorStore.get(ConstructorInjectorStore.java:50)
        at org.elasticsearch.common.inject.ConstructorBindingImpl.initialize(ConstructorBindingImpl.java:50)
        at org.elasticsearch.common.inject.InjectorImpl.initializeBinding(InjectorImpl.java:372)
        at org.elasticsearch.common.inject.BindingProcessor$1$1.run(BindingProcessor.java:148)
        at org.elasticsearch.common.inject.BindingProcessor.initializeBindings(BindingProcessor.java:204)
        at org.elasticsearch.common.inject.InjectorBuilder.initializeStatically(InjectorBuilder.java:119)
        at org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:102)
        at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:93)
        at org.elasticsearch.common.inject.Guice.createInjector(Guice.java:70)
        at org.elasticsearch.common.inject.ModulesBuilder.createInjector(ModulesBuilder.java:59)
        at org.elasticsearch.node.internal.InternalNode.<init>(InternalNode.java:203)
        at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:159)
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:70)
        at org.elasticsearch.bootstrap.Bootstrap.main(Bootstrap.java:213)
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:32)
Caused by: org.elasticsearch.common.util.concurrent.ExecutionError: java.lang.NoClassDefFoundError: Lorg/elasticsearch/indices/InternalIndicesService;
        at org.elasticsearch.common.cache.LocalCache$Segment.get(LocalCache.java:2201)
        at org.elasticsearch.common.cache.LocalCache.get(LocalCache.java:3937)
        at org.elasticsearch.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
        at org.elasticsearch.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
        at org.elasticsearch.common.inject.internal.FailableCache.get(FailableCache.java:51)
        at org.elasticsearch.common.inject.MembersInjectorStore.get(MembersInjectorStore.java:68)
        at org.elasticsearch.common.inject.ConstructorInjectorStore.createConstructor(ConstructorInjectorStore.java:67)
        at org.elasticsearch.common.inject.ConstructorInjectorStore.access$000(ConstructorInjectorStore.java:29)
        at org.elasticsearch.common.inject.ConstructorInjectorStore$1.create(ConstructorInjectorStore.java:37)
        at org.elasticsearch.common.inject.ConstructorInjectorStore$1.create(ConstructorInjectorStore.java:33)
        at org.elasticsearch.common.inject.internal.FailableCache$1.load(FailableCache.java:39)
        at org.elasticsearch.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
        at org.elasticsearch.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
        at org.elasticsearch.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
        at org.elasticsearch.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
        ... 19 more
Caused by: java.lang.NoClassDefFoundError: Lorg/elasticsearch/indices/InternalIndicesService;
        at java.lang.Class.getDeclaredFields0(Native Method)
        at java.lang.Class.privateGetDeclaredFields(Class.java:2499)
        at java.lang.Class.getDeclaredFields(Class.java:1811)
        at org.elasticsearch.common.inject.spi.InjectionPoint$Factory$1.getMembers(InjectionPoint.java:378)
        at org.elasticsearch.common.inject.spi.InjectionPoint$Factory$1.getMembers(InjectionPoint.java:376)
        at org.elasticsearch.common.inject.spi.InjectionPoint.addInjectorsForMembers(InjectionPoint.java:351)
        at org.elasticsearch.common.inject.spi.InjectionPoint.addInjectionPoints(InjectionPoint.java:345)
        at org.elasticsearch.common.inject.spi.InjectionPoint.forInstanceMethodsAndFields(InjectionPoint.java:287)
        at org.elasticsearch.common.inject.MembersInjectorStore.createWithListeners(MembersInjectorStore.java:80)
        at org.elasticsearch.common.inject.MembersInjectorStore.access$000(MembersInjectorStore.java:36)
        at org.elasticsearch.common.inject.MembersInjectorStore$1.create(MembersInjectorStore.java:45)
        at org.elasticsearch.common.inject.MembersInjectorStore$1.create(MembersInjectorStore.java:41)
        at org.elasticsearch.common.inject.internal.FailableCache$1.load(FailableCache.java:39)
        at org.elasticsearch.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
        at org.elasticsearch.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
        at org.elasticsearch.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
        at org.elasticsearch.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
        ... 33 more
Caused by: java.lang.ClassNotFoundException: org.elasticsearch.indices.InternalIndicesService
        at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
        ... 50 more

[tlrx]

@cyberhuman thanks :) It's possible that some installed plugins are not compatible with the version of elasticsearch you use. One thing you can do is to test if elasticsearch starts correctly without plugins. If it starts correctly, then try to install each plugin one by one, restarting elasticsearch after each plugin installation. This should indicates the potential plugin that fails.

Also, note that few plugins have been updated for ES 1.5: analysis-icu, analysis-kuromoji etc. So be sure to use updated plugins.

[cyberhuman]

I was able to nail it down to the marvel plugin. Too bad it does not complain about being incompatible with ES. Thanks for the help anyway!

[tlrx]

@cyberhuman thanks for letting us know :)

[bleskes]

@cyberhuman marvel 1.3.1 should work with 1.5 . Can you double check which version you use(d) ? if not, can you install the lastest marvel and give it a spin? (works for me :))

[cyberhuman]

@bleskes, honestly I don't know. The official download page recommends /latest, and I have already reinstalled it, so there is no way to see know what was the previous version. But the latest version works fine, thanks.

[bleskes]

@cyberhuman if it's works now I'm happy. The latest is a redirect link to the latest version compatible with your ES. You can check the marvel version by looking at the plugins/marvel folder . it will be part of the jar name. Also GET /_cat/plugins will give you versions as well.

[chbrown]

I hit the same issue using Homebrew. After removing the marvel plugin it works just fine, but the error message says nothing about plugin incompatibility, which would have been helpful. Or even better, skip loading plugins that are incompatible, instead of completely failing to start.

[tlrx]

@chbrown Agreed, there's a room for improvement here. Be sure that we have it in mind :)