Not able to transfer data from hive to elastic-search #417
Comments
|
You seem to be confusing a lot of things here. es-hadoop is not a plugin. I'm not sure where you got this idea but if you look at the docs (which I highly recommend you do) you would have noticed that
doesn't do anything. Further more you indicate that you are using elasticsearch-hadoop-hive-2.1.0.Beta3.jar yet install the same library as a plugin. es-hadoop is a connector and should not be installed as a plugin. In fact, it doesn't touch the elasticsearch installation. As explained several times in the docs, it's a jar that you add to your Hadoop job, and not elasticsearch. Second, as mentioned in the issues you posted, you need only one version of es-hadoop - if you have multiple versions (like 2.1.0.Beta3 and 2.1.0-SNAPSHOT) you are bound to run into errors. I highly recommend you spend some time with the docs which indicate how you can install es-hadoop connector. |
|
@costin Thank you for your help. |
|
Hi guys, I am facing the same issue after applying the solution as you guys mention in your post. My ENV is as follows: CentOS 6 I have performed the following steps till now:
2)made entry in hive-site.xml file as below 3)from csv file i have created one table with the following command: create table event(event_time STRING,name STRING,serverity STRING,sourceaddress STRING,categorysignificance STRING) ROW FORMAT DELIMITED FIELDS TERMINATED BY ','; 4)then created external table on the top of ES create external table eventlog(event_time STRING,name STRING,serverity STRING,sourceaddress STRING,categorysignificance STRING) NOTE - I have already created eventlog index in ES and indexed one document into it manually. 5)after creating the external table i am able to read the indexed data from ES using following query: select * from eventlog; 6)but when i try to load the data into ES from table which i have created using csv file using follwing command: insert into table eventlog select * from event; After running the above query i got this error in hive console: Query ID = horton_20150515053434_f30d4a09-afbf-4593-9dc6-3f28c17ae651 Below is the detailed log of the same issue: 2015-05-18 03:07:36,054 INFO [main]: parse.ParseDriver (ParseDriver.java:parse(185)) - Parsing command: insert into table eventlog select * from event 2015-05-18 03:07:46,501 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogBegin(108)) - 2015-05-18 03:07:46,501 INFO [ATS Logger 0]: hooks.ATSHook (ATSHook.java:createPostHookEvent(170)) - Received post-hook notification for :horton_20150518030707_32a637a4-9b82-43ef-9515-32979a634352 NOTE - when i tried by creating external table with this command: create external table sample_ex(event_time STRING,name STRING,serverity STRING,sourceaddress STRING,categorysignificance STRING); Im able to insert the data from event table to this newly created sample_ex table. I want to know what is the cause that the insertion is failing for the external table since its data storage type is ES. and also in detail log i can see the failure of creation of hook for the job what is that. because the hook was created at the time of "sample_ex" external table creation. Thanks for any help and let me know if you want any further information. Regards! |
I want use es-hadoop-hive,My ENV is
I've already seen these issues： #416 #359 #409
hence I git clone source code(branch master) and use gradlew to generate
elasticsearch-hadoop-hive-2.1.0.BUILD-SNAPSHOT.jar
copy this jar file to hive lib and set hive-site.xml
but this plugin is unable to work.
my script is:
node log:
Thanks for any help!!
Best Regards
The text was updated successfully, but these errors were encountered: