Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Could not resolve org.elasticsearch.hadoop.pig.ESStorage #291

Closed
zoo9000 opened this issue Oct 2, 2014 · 1 comment
Closed

Could not resolve org.elasticsearch.hadoop.pig.ESStorage #291

zoo9000 opened this issue Oct 2, 2014 · 1 comment

Comments

@zoo9000
Copy link

zoo9000 commented Oct 2, 2014

Hi,

I downloaded the elasticsearch-hadoop-2.0.1.jar file from
http://www.elasticsearch.org/overview/hadoop/download/

I am trying to run a small pig-script to write some data using ESStorage.
Here is my script,

REGISTER /usr/lib/pig/lib/elasticsearch-hadoop-2.0.1.jar;
A = LOAD '/demo/apache.log' USING PigStorage() AS
(type:chararray, time:chararray, extension:chararray, clientip:chararray, request:chararray, response:long, bytes:long, phpmemory:long, agent:chararray);
B = FOREACH A GENERATE time, type, extension, clientip, request, response, bytes, agent;
STORE B INTO 'demo/pig' USING org.elasticsearch.hadoop.pig.ESStorage('es.mapping.names=time:@timestamp');

My input is on HDFS and is the example apache-log data defined here

http://download.elasticsearch.org/hadoop/short-video-1/apache.zip

I'm running on a standard Hortonworks HDP2.1 sandbox. 
But when I run the script I get this error message (see below).

I've also tried running it with the command-line parameter option
but get the same error,

pig -Dpig.additional.jars=/usr/lib/pig/lib/elasticsearch-hadoop-2.0.1.jar -f script.pig
[root@sandbox CostinLeau]# pig -Dpig.additional.jars=/usr/lib/pig/lib/elasticsearch-hadoop-2.0.1.jar script.pig
2014-10-02 05:46:31,326 [main] INFO org.apache.pig.Main - Apache Pig version 0.12.1.2.1.1.0-237 (rexported) compiled Mar 24 2014, 08:39:05
2014-10-02 05:46:31,326 [main] INFO org.apache.pig.Main - Logging error messages to: /root/Hive-ES/CostinLeau/pig_1412253991324.log
2014-10-02 05:46:32,860 [main] INFO org.apache.pig.impl.util.Utils - Default bootup file /root/.pigbootup not found
2014-10-02 05:46:33,319 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2014-10-02 05:46:33,319 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
2014-10-02 05:46:33,320 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: hdfs://sandbox.hortonworks.com:8020
2014-10-02 05:46:34,762 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
2014-10-02 05:46:35,864 [main] ERROR org.apache.pig.PigServer - exception during parsing: Error during parsing. Could not resolve org.elasticsearch.hadoop.pig.ESStorage using imports: [, java.lang., org.apache.pig.builtin., org.apache.pig.impl.builtin.]
Failed to parse: Pig script failed to parse:
<file script.pig, line 12, column 30> pig script failed to validate: org.apache.pig.backend.executionengine.ExecException: ERROR 1070: Could not resolve org.elasticsearch.hadoop.pig.ESStorage using imports: [, java.lang., org.apache.pig.builtin., org.apache.pig.impl.builtin.]
at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:196)
at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1678)
at org.apache.pig.PigServer$Graph.access$000(PigServer.java:1411)
at org.apache.pig.PigServer.parseAndBuild(PigServer.java:344)
at org.apache.pig.PigServer.executeBatch(PigServer.java:369)
at org.apache.pig.PigServer.executeBatch(PigServer.java:355)
at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:140)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:202)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:173)
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
at org.apache.pig.Main.run(Main.java:607)
at org.apache.pig.Main.main(Main.java:156)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Caused by:
<file script.pig, line 12, column 30> pig script failed to validate: org.apache.pig.backend.executionengine.ExecException: ERROR 1070: Could not resolve org.elasticsearch.hadoop.pig.ESStorage using imports: [, java.lang., org.apache.pig.builtin., org.apache.pig.impl.builtin.]
at org.apache.pig.parser.LogicalPlanBuilder.validateFuncSpec(LogicalPlanBuilder.java:1299)
at org.apache.pig.parser.LogicalPlanBuilder.buildFuncSpec(LogicalPlanBuilder.java:1284)
at org.apache.pig.parser.LogicalPlanGenerator.func_clause(LogicalPlanGenerator.java:5158)
at org.apache.pig.parser.LogicalPlanGenerator.store_clause(LogicalPlanGenerator.java:7756)
at org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.java:1669)
at org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGenerator.java:1102)
at org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.java:560)
at org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:421)
at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:188)
... 16 more
Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 1070: Could not resolve org.elasticsearch.hadoop.pig.ESStorage using imports: [, java.lang., org.apache.pig.builtin., org.apache.pig.impl.builtin.]
at org.apache.pig.impl.PigContext.resolveClassName(PigContext.java:653)
at org.apache.pig.parser.LogicalPlanBuilder.validateFuncSpec(LogicalPlanBuilder.java:1296)
... 24 more
2014-10-02 05:46:35,872 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1070: Could not resolve org.elasticsearch.hadoop.pig.ESStorage using imports: [, java.lang., org.apache.pig.builtin., org.apache.pig.impl.builtin.]
Details at logfile: /root/Hive-ES/CostinLeau/pig_1412253991324.log
[root@sandbox CostinLeau]#

Any help would be appreciated,

Ken

@costin
Copy link
Member

costin commented Oct 3, 2014

@zoo9000 / Ken you have classpath issues which are best addressed through the mailing list or IRC as explained in the docs. Moreover you are not using the correct class (should be EsStorage and not ESStorage - notice the difference in the upper case).

I strongly recommend looking at the documentation and checking the code snippets there.
Also your file paths seem incorrect - you are using Pig yet you're using /root/Hive-ES. Such confusion and details can become hard to debug as time goes by. And unfortunately including my name in the classpath will not solve anything...

P.S. I appreciate the formatting but unfortunately it was applied incorrect as you can see from the code - see the preview tab which helps you visualize it before posting.

@costin costin closed this as completed Oct 3, 2014
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants