Let me first start by saying that partitions and external tables are clunky and buggy.
If I understand correctly you'd like to use the partitions of a table as the type for writing data to Elasticsearch.
This should be fairly easy to achieve by parameterizing your Hive script - create a simple script that reads a table based on partition {X} and writes to Elasticsearch based on index/{X}
Then run the Hive script binding {X} to the partition you desire.

As a side note, partitioning is typically used to improve performance - if you deal with large volumes of data you probably want each partition to point to a different index instead of a different type (since that would push all the data under the same index).

" parameterizing your Hive script " This is what I am currently doing. The issue is that I cannot query multiple partitions at the same time.

I also see lots of connection(out of nodes and retry?) exceptions in my mapper. Is there any known tune up I could do to avoid this issue? i have tried to increase the time out to be 10m, it seems to get better, but wanna know if any better ways.

My cluster has 6 machines each ES is running with 20G mem, and each partition is around 30million records.(with 4-5 string fields).

From an elasticsearch perspective, if you push your data under the same index, you can access it with the same query. If you have multiple indices, you can create a query that queries all of them.

The connection exceptions can have a plethora of reasons and without any concrete information I can only guess what's the issue. If you are writing then consider minimizing the bulk size (we'll do this in the next release); if it's reading then depends on how you stream data.

You can always turn on logging on the various packages in org.elasticsearch.hadoop to see what's wrong. I also recommend trying the latest master.

Thanks Costin. All my operations are writes. I will try to decrease the bulk size, and will turn on the logging. will keep you posted.

Try with a bulk size of 5 MBs and move from there. Note that, this is the batch size per task - if you job has 10 jobs, it leads to 100MBs bulks, 20, 200MBs, etc...