Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
Insert into elastic search from a partitioned table throws error #724
I have noticed that Selecting data from a partitioned hive table and inserting into elastic search does not work very will and the map reduce job ends in the following error.
I have tested similar scenarios by using the different source tables ( Stored as parquet, stored as parquet and snappy compressed) and it works fine. But when i use partitioned hive table as my source table, the job fails with the above error.
I have used Cloudera 5.5 VM for hadoop, elasticsearch-2.2.1 and elasticsearch-hadoop-2.2.0-rc1.jar for my tests.
I attach a zip file with two HQL scripts and the ES-Hadoop jar for reproducing this issue.
Thanks and Regards
I am facing exactly the same issue. Difficult to incriminate ES-Hadoop indeed since the issue occurs only with partitioned tables in Parquet format (no issue with a flat tables in Parquet format or partitioned table in Avro format). Very likely that the issue is in Parquet Serde, nothing to do with ES-Hadoop
My solution is create temp table . Maybe the issue is in
Above 2 step test pass .... This exception doesn't happen.