fscrawler does not recover when it lost communication with elasticsearch

[twindragons1987]

Hi
I am trying to run fscrawler on my local machine to read data that has been extracted from readpst command and sending it for elasticsearh. It works fine for some iterations and then it gives following error and doesn't recover. Can some one suggest if I am doing something wrong

17:06:22,255 DEBUG [f.p.e.c.f.FsCrawlerImpl] Looking for removed files in [/MY_PATH/]...
17:06:22,255 TRACE [f.p.e.c.f.FsCrawlerImpl] Querying elasticsearch for files in dir [path.encoded:7fbc1f13665e5067aa6550ad5b4a6ba5]
17:06:22,255 WARN  [f.p.e.c.f.FsCrawlerImpl] Error while indexing content from [java.io.IOException: no active node found. Start an elasticsearch cluster first! Expecting something running at [localhost:9200], /MY_PATH]
17:06:22,255 DEBUG [f.p.e.c.f.FsCrawlerImpl] Fs crawler is going to sleep for 15m
17:06:26,295 DEBUG [f.p.e.c.f.c.BulkProcessor] Going to execute new bulk composed of 14 actions
17:06:26,295 WARN  [f.p.e.c.f.c.BulkProcessor] Error executing bulk
java.io.IOException: no active node found. Start an elasticsearch cluster first! Expecting something running at [localhost:9200]
at fr.pilato.elasticsearch.crawler.fs.client.ElasticsearchClient.findNextNode(ElasticsearchClient.java:114) ~[fscrawler-2.1.jar:?]
	at fr.pilato.elasticsearch.crawler.fs.client.ElasticsearchClient.bulk(ElasticsearchClient.java:243) ~[fscrawler-2.1.jar:?]
	at fr.pilato.elasticsearch.crawler.fs.client.BulkProcessor.execute(BulkProcessor.java:136) ~[fscrawler-2.1.jar:?]
	at fr.pilato.elasticsearch.crawler.fs.client.BulkProcessor.executeWhenNeeded(BulkProcessor.java:123) ~[fscrawler-2.1.jar:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_111]
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_111]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_111]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_111]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_111]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_111]
	at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]

[dadoonet]

In 2.2, I changed from internal home made REST client to the official REST client (#203).
It might have a better exception handling.

Any chance you can try the 2.2-SNAPSHOT version?

[dadoonet]

Closing for now. Feel free to reopen if you can see that problem again with 2.2.

[twindragons1987]

Can you please provide me with a link to 2.2?

[dadoonet]

Sure. The README gives it:

From https://oss.sonatype.org/content/repositories/snapshots/fr/pilato/elasticsearch/crawler/fscrawler/2.2-SNAPSHOT/
Latest build: https://oss.sonatype.org/content/repositories/snapshots/fr/pilato/elasticsearch/crawler/fscrawler/2.2-SNAPSHOT/fscrawler-2.2-20170104.191528-56.zip

HTH

[c4tom]

I have, maybe, same problem (i am using last version)

09:21:45,826 WARN  [f.p.e.c.f.c.BulkProcessor] Error executing bulk
java.io.IOException: listener timeout after waiting for [30000] ms
	at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:617) ~[jar:rsrc:rest-5.2.2.jar!/:?]
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:212) ~[jar:rsrc:rest-5.2.2.jar!/:?]
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:184) ~[jar:rsrc:rest-5.2.2.jar!/:?]
	at fr.pilato.elasticsearch.crawler.fs.client.ElasticsearchClient.bulk(ElasticsearchClient.java:149) ~[rsrc:./:?]
	at fr.pilato.elasticsearch.crawler.fs.client.BulkProcessor.execute(BulkProcessor.java:157) [rsrc:./:?]
	at fr.pilato.elasticsearch.crawler.fs.client.BulkProcessor.executeIfNeeded(BulkProcessor.java:137) [rsrc:./:?]
	at fr.pilato.elasticsearch.crawler.fs.client.BulkProcessor.internalAdd(BulkProcessor.java:130) [rsrc:./:?]
	at fr.pilato.elasticsearch.crawler.fs.client.BulkProcessor.add(BulkProcessor.java:118) [rsrc:./:?]
	at fr.pilato.elasticsearch.crawler.fs.client.BulkProcessor.add(BulkProcessor.java:86) [rsrc:./:?]
	at fr.pilato.elasticsearch.crawler.fs.FsCrawlerImpl$FSParser.esIndex(FsCrawlerImpl.java:719) [rsrc:./:?]
	at fr.pilato.elasticsearch.crawler.fs.FsCrawlerImpl$FSParser.esIndex(FsCrawlerImpl.java:703) [rsrc:./:?]
	at fr.pilato.elasticsearch.crawler.fs.FsCrawlerImpl$FSParser.indexFile(FsCrawlerImpl.java:596) [rsrc:./:?]
	at fr.pilato.elasticsearch.crawler.fs.FsCrawlerImpl$FSParser.addFilesRecursively(FsCrawlerImpl.java:386) [rsrc:./:?]
	at fr.pilato.elasticsearch.crawler.fs.FsCrawlerImpl$FSParser.addFilesRecursively(FsCrawlerImpl.java:405) [rsrc:./:?]
	at fr.pilato.elasticsearch.crawler.fs.FsCrawlerImpl$FSParser.run(FsCrawlerImpl.java:273) [rsrc:./:?]
	at java.lang.Thread.run(Thread.java:745) [?:1.8.0_66]

[dadoonet]

@candido1212 In which context is this happening? FSCrawler was running and was able to index then suddenly started to fail?
Anything in elasticsearch logs?

If elasticsearch restarts, is FSCrawler able to recover?

Would help a lot if you can provide more details. Feel free to open another issue.