Too many requests?

[AlexeyRaga]

I am trying to save documents from Spark in a monthly index using:

.saveToEs("{index:yyyy-MM}/myDoc")

But unfortunately I always get the following error:

Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 1.0 failed 1 times, most recent failure: Lost task 0.0 in stage 1.0 (TID 1, localhost): org.elasticsearch.hadoop.rest.EsHadoopInvalidRequest: Found unrecoverable error [Too Many Requests(429) 
- [rejected execution of org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryPhase$1@15d274d6 
    on EsThreadPoolExecutor[bulk, queue capacity = 50, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@3acacba8[
    Running, pool size = 8, active threads = 8, queued tasks = 50, completed tasks = 3955]]]]; 
Bailing out..

If I use the same index all the time then it works fine.

What would be the correct way of doing this?

[costin]

Can you provide some info regarding your environment - what version of ES and ES-Hadoop are you using?
Also how many indices are created? Does the exception appear right away or after some time?

[AlexeyRaga]

After some time.
I figured it out that if I increase the queue capacity (say, to 1000) the exception disappears, so I'll close this issue.
But it is really strange that it only happened when using partitioned indices.