NUTCH-2287 Indexer-elastic plugin should use Elasticsearch BulkProcessor and BackoffPolicy

[naegelejd]

No description provided.

[lewismc]

I am +1 on this and will merge shortly unless objections

[naegelejd]

Thanks Lewis. Are any active maintainers using Elasticsearch? It may be good if someone (anyone) gives it a quick try.

[knap1930]

I get the following exception in the CleaningJob

Error: java.lang.IllegalStateException: bulk process already closed
        at org.elasticsearch.action.bulk.BulkProcessor.ensureOpen(BulkProcessor.java:278)
        at org.elasticsearch.action.bulk.BulkProcessor.flush(BulkProcessor.java:329)
        at org.apache.nutch.indexwriter.elastic.ElasticIndexWriter.commit(ElasticIndexWriter.java:198)
        at org.apache.nutch.indexer.IndexWriters.commit(IndexWriters.java:125)
        at org.apache.nutch.indexer.CleaningJob$DeleterReducer.close(CleaningJob.java:123)
        at org.apache.hadoop.io.IOUtils.cleanup(IOUtils.java:244)
        at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:467)
        at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:393)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)