You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I am trying to run a small pig-script to write some data using ESStorage.
Here is my script,
REGISTER /usr/lib/pig/lib/elasticsearch-hadoop-2.0.1.jar;
A = LOAD '/demo/apache.log' USING PigStorage() AS
(type:chararray, time:chararray, extension:chararray, clientip:chararray, request:chararray, response:long, bytes:long, phpmemory:long, agent:chararray);
B = FOREACH A GENERATE time, type, extension, clientip, request, response, bytes, agent;
STORE B INTO 'demo/pig' USING org.elasticsearch.hadoop.pig.ESStorage('es.mapping.names=time:@timestamp');
My input is on HDFS and is the example apache-log data defined here
I'm running on a standard Hortonworks HDP2.1 sandbox.
But when I run the script I get this error message (see below).
I've also tried running it with the command-line parameter option
but get the same error,
pig -Dpig.additional.jars=/usr/lib/pig/lib/elasticsearch-hadoop-2.0.1.jar -f script.pig
[root@sandbox CostinLeau]# pig -Dpig.additional.jars=/usr/lib/pig/lib/elasticsearch-hadoop-2.0.1.jar script.pig
2014-10-02 05:46:31,326 [main] INFO org.apache.pig.Main - Apache Pig version 0.12.1.2.1.1.0-237 (rexported) compiled Mar 24 2014, 08:39:05
2014-10-02 05:46:31,326 [main] INFO org.apache.pig.Main - Logging error messages to: /root/Hive-ES/CostinLeau/pig_1412253991324.log
2014-10-02 05:46:32,860 [main] INFO org.apache.pig.impl.util.Utils - Default bootup file /root/.pigbootup not found
2014-10-02 05:46:33,319 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2014-10-02 05:46:33,319 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
2014-10-02 05:46:33,320 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: hdfs://sandbox.hortonworks.com:8020
2014-10-02 05:46:34,762 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
2014-10-02 05:46:35,864 [main] ERROR org.apache.pig.PigServer - exception during parsing: Error during parsing. Could not resolve org.elasticsearch.hadoop.pig.ESStorage using imports: [, java.lang., org.apache.pig.builtin., org.apache.pig.impl.builtin.]
Failed to parse: Pig script failed to parse:
<file script.pig, line 12, column 30> pig script failed to validate: org.apache.pig.backend.executionengine.ExecException: ERROR 1070: Could not resolve org.elasticsearch.hadoop.pig.ESStorage using imports: [, java.lang., org.apache.pig.builtin., org.apache.pig.impl.builtin.]
at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:196)
at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1678)
at org.apache.pig.PigServer$Graph.access$000(PigServer.java:1411)
at org.apache.pig.PigServer.parseAndBuild(PigServer.java:344)
at org.apache.pig.PigServer.executeBatch(PigServer.java:369)
at org.apache.pig.PigServer.executeBatch(PigServer.java:355)
at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:140)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:202)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:173)
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
at org.apache.pig.Main.run(Main.java:607)
at org.apache.pig.Main.main(Main.java:156)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Caused by:
<file script.pig, line 12, column 30> pig script failed to validate: org.apache.pig.backend.executionengine.ExecException: ERROR 1070: Could not resolve org.elasticsearch.hadoop.pig.ESStorage using imports: [, java.lang., org.apache.pig.builtin., org.apache.pig.impl.builtin.]
at org.apache.pig.parser.LogicalPlanBuilder.validateFuncSpec(LogicalPlanBuilder.java:1299)
at org.apache.pig.parser.LogicalPlanBuilder.buildFuncSpec(LogicalPlanBuilder.java:1284)
at org.apache.pig.parser.LogicalPlanGenerator.func_clause(LogicalPlanGenerator.java:5158)
at org.apache.pig.parser.LogicalPlanGenerator.store_clause(LogicalPlanGenerator.java:7756)
at org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.java:1669)
at org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGenerator.java:1102)
at org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.java:560)
at org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:421)
at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:188)
... 16 more
Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 1070: Could not resolve org.elasticsearch.hadoop.pig.ESStorage using imports: [, java.lang., org.apache.pig.builtin., org.apache.pig.impl.builtin.]
at org.apache.pig.impl.PigContext.resolveClassName(PigContext.java:653)
at org.apache.pig.parser.LogicalPlanBuilder.validateFuncSpec(LogicalPlanBuilder.java:1296)
... 24 more
2014-10-02 05:46:35,872 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1070: Could not resolve org.elasticsearch.hadoop.pig.ESStorage using imports: [, java.lang., org.apache.pig.builtin., org.apache.pig.impl.builtin.]
Details at logfile: /root/Hive-ES/CostinLeau/pig_1412253991324.log
[root@sandbox CostinLeau]#
Any help would be appreciated,
Ken
The text was updated successfully, but these errors were encountered:
@zoo9000 / Ken you have classpath issues which are best addressed through the mailing list or IRC as explained in the docs. Moreover you are not using the correct class (should be EsStorage and notESStorage - notice the difference in the upper case).
I strongly recommend looking at the documentation and checking the code snippets there.
Also your file paths seem incorrect - you are using Pig yet you're using /root/Hive-ES. Such confusion and details can become hard to debug as time goes by. And unfortunately including my name in the classpath will not solve anything...
P.S. I appreciate the formatting but unfortunately it was applied incorrect as you can see from the code - see the preview tab which helps you visualize it before posting.
Hi,
I downloaded the elasticsearch-hadoop-2.0.1.jar file from
http://www.elasticsearch.org/overview/hadoop/download/
REGISTER /usr/lib/pig/lib/elasticsearch-hadoop-2.0.1.jar;
A = LOAD '/demo/apache.log' USING PigStorage() AS
(type:chararray, time:chararray, extension:chararray, clientip:chararray, request:chararray, response:long, bytes:long, phpmemory:long, agent:chararray);
B = FOREACH A GENERATE time, type, extension, clientip, request, response, bytes, agent;
STORE B INTO 'demo/pig' USING org.elasticsearch.hadoop.pig.ESStorage('es.mapping.names=time:@timestamp');
http://download.elasticsearch.org/hadoop/short-video-1/apache.zip
pig -Dpig.additional.jars=/usr/lib/pig/lib/elasticsearch-hadoop-2.0.1.jar -f script.pig
[root@sandbox CostinLeau]# pig -Dpig.additional.jars=/usr/lib/pig/lib/elasticsearch-hadoop-2.0.1.jar script.pig
2014-10-02 05:46:31,326 [main] INFO org.apache.pig.Main - Apache Pig version 0.12.1.2.1.1.0-237 (rexported) compiled Mar 24 2014, 08:39:05
2014-10-02 05:46:31,326 [main] INFO org.apache.pig.Main - Logging error messages to: /root/Hive-ES/CostinLeau/pig_1412253991324.log
2014-10-02 05:46:32,860 [main] INFO org.apache.pig.impl.util.Utils - Default bootup file /root/.pigbootup not found
2014-10-02 05:46:33,319 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2014-10-02 05:46:33,319 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
2014-10-02 05:46:33,320 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: hdfs://sandbox.hortonworks.com:8020
2014-10-02 05:46:34,762 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
2014-10-02 05:46:35,864 [main] ERROR org.apache.pig.PigServer - exception during parsing: Error during parsing. Could not resolve org.elasticsearch.hadoop.pig.ESStorage using imports: [, java.lang., org.apache.pig.builtin., org.apache.pig.impl.builtin.]
Failed to parse: Pig script failed to parse:
<file script.pig, line 12, column 30> pig script failed to validate: org.apache.pig.backend.executionengine.ExecException: ERROR 1070: Could not resolve org.elasticsearch.hadoop.pig.ESStorage using imports: [, java.lang., org.apache.pig.builtin., org.apache.pig.impl.builtin.]
at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:196)
at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1678)
at org.apache.pig.PigServer$Graph.access$000(PigServer.java:1411)
at org.apache.pig.PigServer.parseAndBuild(PigServer.java:344)
at org.apache.pig.PigServer.executeBatch(PigServer.java:369)
at org.apache.pig.PigServer.executeBatch(PigServer.java:355)
at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:140)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:202)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:173)
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
at org.apache.pig.Main.run(Main.java:607)
at org.apache.pig.Main.main(Main.java:156)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Caused by:
<file script.pig, line 12, column 30> pig script failed to validate: org.apache.pig.backend.executionengine.ExecException: ERROR 1070: Could not resolve org.elasticsearch.hadoop.pig.ESStorage using imports: [, java.lang., org.apache.pig.builtin., org.apache.pig.impl.builtin.]
at org.apache.pig.parser.LogicalPlanBuilder.validateFuncSpec(LogicalPlanBuilder.java:1299)
at org.apache.pig.parser.LogicalPlanBuilder.buildFuncSpec(LogicalPlanBuilder.java:1284)
at org.apache.pig.parser.LogicalPlanGenerator.func_clause(LogicalPlanGenerator.java:5158)
at org.apache.pig.parser.LogicalPlanGenerator.store_clause(LogicalPlanGenerator.java:7756)
at org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.java:1669)
at org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGenerator.java:1102)
at org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.java:560)
at org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:421)
at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:188)
... 16 more
Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 1070: Could not resolve org.elasticsearch.hadoop.pig.ESStorage using imports: [, java.lang., org.apache.pig.builtin., org.apache.pig.impl.builtin.]
at org.apache.pig.impl.PigContext.resolveClassName(PigContext.java:653)
at org.apache.pig.parser.LogicalPlanBuilder.validateFuncSpec(LogicalPlanBuilder.java:1296)
... 24 more
2014-10-02 05:46:35,872 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1070: Could not resolve org.elasticsearch.hadoop.pig.ESStorage using imports: [, java.lang., org.apache.pig.builtin., org.apache.pig.impl.builtin.]
Details at logfile: /root/Hive-ES/CostinLeau/pig_1412253991324.log
[root@sandbox CostinLeau]#
Any help would be appreciated,
Ken
The text was updated successfully, but these errors were encountered: