Switched back to using 1.35, but getting Lucene NoSuchDirectoryException: pointing at directory under 1.4 install now deleted.

[Analect]

My ES installation is just a single node on localhost.
I've been experimenting (more like grinding teeth) with both mongodb_river and mongo-connector to try to get attachments/files indexed into ES. All this was previously working with ES 1.3.1 and Mongo 2.4.11. Tried getting it running using a ES 1.4.0 / Mongo 2.6.5 combination, but facing same issues as reported here: richardwilly98/elasticsearch-river-mongodb#412
Then tried moving back to ES 1.3.5 with requisite installs for mapper-attachment and river_mongodb, but now not even able to open the 'head' plugin (although I can see that ES has started). Part of the error is reproduced below. Why would an ES 1.3.5 installation reference a _river from my ES 1.4.0 data directory? I shut down the mongodb replicaset thinking that maybe that was the source of the problem ... but ES 1.3.5 is still not available to the head plugin because of this lucene NoSuchDirectoryException.

Any thoughts? Thanks.

[2014-11-20 04:38:22,503][DEBUG][action.admin.cluster.node.stats] [Diamond Lil] failed to execute on node [V3oubzV6THmie2DR5LSReg]
org.elasticsearch.transport.RemoteTransportException: [DJ][inet[/10.211.55.6:9300]][cluster/nodes/stats/n]
Caused by: org.elasticsearch.ElasticsearchException: io exception while building 'store stats'
at org.elasticsearch.index.shard.service.InternalIndexShard.storeStats(InternalIndexShard.java:542)
at org.elasticsearch.action.admin.indices.stats.CommonStats.(CommonStats.java:134)
at org.elasticsearch.action.admin.indices.stats.ShardStats.(ShardStats.java:49)
at org.elasticsearch.indices.InternalIndicesService.stats(InternalIndicesService.java:212)
at org.elasticsearch.node.service.NodeService.stats(NodeService.java:156)
at org.elasticsearch.action.admin.cluster.node.stats.TransportNodesStatsAction.nodeOperation(TransportNodesStatsAction.java:96)
at org.elasticsearch.action.admin.cluster.node.stats.TransportNodesStatsAction.nodeOperation(TransportNodesStatsAction.java:44)
at org.elasticsearch.action.support.nodes.TransportNodesOperationAction$NodeTransportHandler.messageReceived(TransportNodesOperationAction.java:278)
at org.elasticsearch.action.support.nodes.TransportNodesOperationAction$NodeTransportHandler.messageReceived(TransportNodesOperationAction.java:269)
at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.lucene.store.NoSuchDirectoryException: directory '/home/mccoole/Development/Tools/ES/elasticsearch-1.4.0/data/elasticsearch/nodes/0/indices/_river/0/index' does not exist
at org.apache.lucene.store.FSDirectory.listAll(FSDirectory.java:218)
at org.apache.lucene.store.FSDirectory.listAll(FSDirectory.java:242)
at org.apache.lucene.store.FileSwitchDirectory.listAll(FileSwitchDirectory.java:87)
at org.apache.lucene.store.FilterDirectory.listAll(FilterDirectory.java:48)
at org.elasticsearch.index.store.DistributorDirectory.listAll(DistributorDirectory.java:88)
at org.apache.lucene.store.FilterDirectory.listAll(FilterDirectory.java:48)
at org.elasticsearch.common.lucene.Directories.estimateSize(Directories.java:40)
at org.elasticsearch.index.store.Store.stats(Store.java:213)
at org.elasticsearch.index.shard.service.InternalIndexShard.storeStats(InternalIndexShard.java:540)

[ZeAleks]

Hello Analect,

I'm facing some performance issue with 1.4 version of Elasticsearch. So I was wondering if I should rollback the installation.
But I red that : http://www.elasticsearch.org/guide/en/elasticsearch/reference/1.4/indices-upgrade.html
1.4 is using new version of lucene. So, maybe, new documents that you inserted has new formats and cannot be used anymore with 1.3 version of Elasticsearch.

If this is the reason for your exception, you have two choices :

• run 1.4 and try to correct what can be
• reindex everything

Keep us in touch.

Good luck.
Alek

[Analect]

@ZeAleks
Thanks for your response.
I don't think I'm affected by what you describe ... as I'm even having this issue on a blank database when I first try to start up a version of ES. I'm not even trying to move around an existing database between versions ... which is why I'm puzzled by why one version of ES (ie. 1.3.5) would want to be referencing a _river that had been used in testing another version (1.4.0). The only explanation I can think of is that it somehow pulled down some reference to the river from a running mongodb replica-set, but I don't know enough about how the mechanism works to be sure if that is a possibility. Even with that replicaset closed down, the same Exception was occurring on the startup of version 1.3.5, where the 'head' plugin is left hanging.

[s1monw]

@Analect do you have the 1.4 instance still running by any chance?

@ZeAleks what kind of perf issues are you seeing?

[ZeAleks]

@s1monw Here is the corresponding post for my perf issues : #8553

[Analect]

@s1monw No, I make sure I only have a single version running at any time to avoid any possibility of conflicts. For the 1.4 version, because I was having a problem getting that running again (I can't recall the detail .. but it might have been something related to the river_mongodb too), I ended up deleting the elasticsearch folder under /data, in expectation that this would clear things up and allow me to start with a new blank database. It obviously hadn't the desired effect ... and then I had this related problem in 1.3.5, where it was looking for a _river definition in that folder I had deleted under the 1.4 installation ... eventhough I had only installed 1.3.5 and hadn't even added any mappings to it yet ... and certainly not any _river mappings.

[Analect]

@s1monw This is odd.
I had completely removed ES 1.4.0 from my machine
I downloaded a new version of the tar and expanded it.
The ES starts fine:
{
"status" : 200,
"name" : "Mr. Justice",
"cluster_name" : "elasticsearch",
"version" : {
"number" : "1.4.0",
"build_hash" : "bc94bd81298f81c656893ab1ddddd30a99356066",
"build_timestamp" : "2014-11-05T14:26:12Z",
"build_snapshot" : false,
"lucene_version" : "4.10.2"
},
"tagline" : "You Know, for Search"
}

However, 'head' is not working ... for this reason, it appears:
mobz/elasticsearch-head#170

What I find really odd though, is that when I try to navigate to 'head' (not yet fixing it as per the link above), I still get a reference to a _river (see last line of logs below) ... eventhough on this new installation of version 1.4, the only plugin I have installed is 'head'.

[2014-11-20 08:06:13,551][INFO ][node ] [Louise Mason] version[1.4.0], pid[8512], build[bc94bd8/2014-11-05T14:26:12Z]
[2014-11-20 08:06:13,552][INFO ][node ] [Louise Mason] initializing ...
[2014-11-20 08:06:13,556][INFO ][plugins ] [Louise Mason] loaded [], sites [head]
[2014-11-20 08:06:15,541][INFO ][node ] [Louise Mason] initialized
[2014-11-20 08:06:15,542][INFO ][node ] [Louise Mason] starting ...
[2014-11-20 08:06:15,631][INFO ][transport ] [Louise Mason] bound_address {inet[/0:0:0:0:0:0:0:0:9301]}, publish_address {inet[/10.211.55.6:9301]}
[2014-11-20 08:06:15,642][INFO ][discovery ] [Louise Mason] elasticsearch/jNaGt-FdSLOtJzOrzbVF8A
[2014-11-20 08:06:18,685][INFO ][cluster.service ] [Louise Mason] detected_master [DJ][V3oubzV6THmie2DR5LSReg][ubuntu][inet[/10.211.55.6:9300]], added {[DJ][V3oubzV6THmie2DR5LSReg][ubuntu][inet[/10.211.55.6:9300]],}, reason: zen-disco-receive(from master [[DJ][V3oubzV6THmie2DR5LSReg][ubuntu][inet[/10.211.55.6:9300]]])
[2014-11-20 08:06:18,721][INFO ][http ] [Louise Mason] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/10.211.55.6:9200]}
[2014-11-20 08:06:18,722][INFO ][node ] [Louise Mason] started
[2014-11-20 08:06:25,859][DEBUG][action.admin.cluster.node.stats] [Louise Mason] failed to execute on node [V3oubzV6THmie2DR5LSReg]
org.elasticsearch.transport.RemoteTransportException: [DJ][inet[/10.211.55.6:9300]][cluster:monitor/nodes/stats[n]]
Caused by: org.elasticsearch.ElasticsearchException: io exception while building 'store stats'
at org.elasticsearch.index.shard.service.InternalIndexShard.storeStats(InternalIndexShard.java:542)
at org.elasticsearch.action.admin.indices.stats.CommonStats.(CommonStats.java:134)
at org.elasticsearch.action.admin.indices.stats.ShardStats.(ShardStats.java:49)
at org.elasticsearch.indices.InternalIndicesService.stats(InternalIndicesService.java:212)
at org.elasticsearch.node.service.NodeService.stats(NodeService.java:156)
at org.elasticsearch.action.admin.cluster.node.stats.TransportNodesStatsAction.nodeOperation(TransportNodesStatsAction.java:96)
at org.elasticsearch.action.admin.cluster.node.stats.TransportNodesStatsAction.nodeOperation(TransportNodesStatsAction.java:44)
at org.elasticsearch.action.support.nodes.TransportNodesOperationAction$NodeTransportHandler.messageReceived(TransportNodesOperationAction.java:278)
at org.elasticsearch.action.support.nodes.TransportNodesOperationAction$NodeTransportHandler.messageReceived(TransportNodesOperationAction.java:269)
at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.lucene.store.NoSuchDirectoryException: directory '/home/mccoole/Development/Tools/ES/elasticsearch-1.4.0/data/elasticsearch/nodes/0/indices/_river/0/index' does not exist

[Analect]

@s1monw
FYI, I made the fix for 'head', as per issue 170 linked above. However, head still won't work and so it might be related to the NoSuchDirectoryException I'm having. Am I right in understanding that there's a separate lucene instance associated with each ES database?

[Analect]

@s1monw
Any suggestions how I can fix this org.apache.lucene.store.NoSuchDirectoryException. I'm not concerned with losing data, as this was a blank database ... I just want to be able to get ES 1.4 up and running again and visible on head ... but somehow (I think!) this directory exception is preventing that on my linux machine. As explained above, I've tried a fresh install, but that doesn't appear to fix, as it still searches for this rogue _river/0/index directory that was deleted long ago. On my windows machine, the 1.4 install with the CORS fix in the *.yml allows me to get head working, so I'm stumped as to why I can't get the linux one working.

[clintongormley]

@Analect you are definitely running more than one node:

[2014-11-20 08:06:18,685][INFO ][cluster.service ] [Louise Mason] detected_master [DJ]

and they're running on the same machine. You can see that this new node started with ports 9301/9201, not 9300/9200

I suggest you do a killall java and start again :)