Can not create a Path from an empty string (Hive MapRed job with ES)

[marcelopaesrech]

When I try to run a mapred job by Hive.
Environment Mac OS or Linux, hive 0.11, hadoop 1.2 or 0.23, ES 0.90.2.

My "test" entity hava just three fields (userid, name and type), and have just one record on ES.

hive> select count(userid) from test;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=
In order to set a constant number of reducers:
set mapred.reduce.tasks=
Starting Job = job_201307081651_0001, Tracking URL = http://localhost:50030/jobdetails.jsp?jobid=job_201307081651_0001
Kill Command = /Users/developergvt/servers/hadoop-1.2.0/libexec/../bin/hadoop job -kill job_201307081651_0001
Hadoop job information for Stage-1: number of mappers: 5; number of reducers: 1
2013-07-08 16:57:39,056 Stage-1 map = 0%, reduce = 0%
2013-07-08 16:58:01,154 Stage-1 map = 100%, reduce = 100%
Ended Job = job_201307081651_0001 with errors
Error during job, obtaining debugging information...
Job Tracking URL: http://localhost:50030/jobdetails.jsp?jobid=job_201307081651_0001
Examining task ID: task_201307081651_0001_m_000006 (and more) from job job_201307081651_0001

Task with the most failures(4):

Task ID:
task_201307081651_0001_m_000000

URL:

http://localhost:50030/taskdetails.jsp?jobid=job_201307081651_0001&tipid=task_201307081651_0001_m_000000

Diagnostic Messages for this Task:
java.lang.IllegalArgumentException: Can not create a Path from an empty string
at org.apache.hadoop.fs.Path.checkPathArg(Path.java:82)
at org.apache.hadoop.fs.Path.(Path.java:90)
at org.apache.hadoop.hive.ql.io.HiveInputFormat$HiveInputSplit.getPath(HiveInputFormat.java:106)
at org.apache.hadoop.mapred.MapTask.updateJobWithSplit(MapTask.java:451)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:409)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:366)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
at org.apache.hadoop.mapred.Child.main(Child.java:249)

FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask
MapReduce Jobs Launched:
Job 0: Map: 5 Reduce: 1 HDFS Read: 0 HDFS Write: 0 FAIL
Total MapReduce CPU Time Spent: 0 msec

[marcelopaesrech]

In my debugs, the Hive doesn't considers the ESInputFormat. HiveInputFormat extends FileInputFormat, and must to implements a getPath. This getPath method, when the internal InputFormat (the internal InputFormat I think is the correct InputFormat to use by Hive) is not a instanceof FileInputFormat, then the getPath returns "". This is throwing the exception. I don't if need another configuration to ES works with Hive, the configuration mentioned in the docs was made.

[marcelopaesrech]

If someone has a resolution for this (maybe configuration, or fix in source code, or even workaround).

[costin]

Hi Marcelo,

Thanks for the follow-up. I'm currently on parental leave but I'll be back 'in the office' on Monday and will look into the issue.
Cheers

[marcelopaesrech]

Ok Costin. Thanks.

[costin]

@marcelopaesrech Can you please share your "test" mapping in ES and table definition in Hive? Thanks.

[marcelopaesrech]

I changed the mapping to one more simple. The mapping of entitie is:
{
"user": {
"properties": {
"username": {
"type": "string"
}
}
}
}
The DDL in hive is:
CREATE EXTERNAL TABLE test
(username STRING)
STORED BY 'org.elasticsearch.hadoop.hive.ESStorageHandler'
TBLPROPERTIES('es.resource'='samples/user/_search?q=*', 'es.host'='localhost', 'es.port'='9200');

And select query is:
select count(username) from test;

And the result is:
hive> select count(username) from test;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=
In order to set a constant number of reducers:
set mapred.reduce.tasks=
Starting Job = job_201307160951_0001, Tracking URL = http://localhost:50030/jobdetails.jsp?jobid=job_201307160951_0001
Kill Command = /Users/developergvt/servers/hadoop-1.0.4/libexec/../bin/hadoop job -kill job_201307160951_0001
Hadoop job information for Stage-1: number of mappers: 5; number of reducers: 1
2013-07-16 10:02:38,591 Stage-1 map = 0%, reduce = 0%
2013-07-16 10:03:08,821 Stage-1 map = 100%, reduce = 100%
Ended Job = job_201307160951_0001 with errors
Error during job, obtaining debugging information...
Job Tracking URL: http://localhost:50030/jobdetails.jsp?jobid=job_201307160951_0001
Examining task ID: task_201307160951_0001_m_000006 (and more) from job job_201307160951_0001

Task with the most failures(4):

Task ID:
task_201307160951_0001_m_000001

URL:

http://localhost:50030/taskdetails.jsp?jobid=job_201307160951_0001&tipid=task_201307160951_0001_m_000001

Diagnostic Messages for this Task:
java.lang.IllegalArgumentException: Can not create a Path from an empty string
at org.apache.hadoop.fs.Path.checkPathArg(Path.java:82)
at org.apache.hadoop.fs.Path.(Path.java:90)
at org.apache.hadoop.hive.ql.io.HiveInputFormat$HiveInputSplit.getPath(HiveInputFormat.java:106)
at org.apache.hadoop.mapred.MapTask.updateJobWithSplit(MapTask.java:453)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:415)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
at org.apache.hadoop.mapred.Child.main(Child.java:249)

FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask
MapReduce Jobs Launched:
Job 0: Map: 5 Reduce: 1 HDFS Read: 0 HDFS Write: 0 FAIL
Total MapReduce CPU Time Spent: 0 msec

[costin]

Thanks - it looks like the count() operator is causing a different code path to be picked up. I'll look into it and report back.

[marcelopaesrech]

I took a look in Hive implementation, and when this job executes, Hive uses a CombineHiveInputFormat that implements FileInputFormat. This CombineHiveInput, apparently, works like a proxy between Hadoop and ESInputFormat. Hadoop calls getPath implemented by CombineHiveInputFormat, and then, CombineHiveInputFormat verifies if ESInputFormat implements FileInputFormat (of course that does not implement), then CombineHiveInputFormat tries to create a new Path(""), and then the Exception occurs.

I looked into HBaseStorageHandler implementation too, but I did't see where thats differs.

I don't think thats just with count operator. Simple queries like "select * from test" work fine. But any query that tries to use MapReduce fails.

[costin]

@marcelopaesrech I've fixed this in master - please take the next snapshot build for a spin and report back your findings. The issue was caused by the fact that when an operator is used, Hive triggers additional tasks (through the operator plan) to be executed. Unfortunately as you've seen, the code base basically expects every Hive resource to be backed by a resource on HDFS - and if this is not the case, incorrect code is executed (as the invalid Path you mentioned).
This is now fixed and hopefully should work across Hive versions (0.10 and 0.11) as well.

P.S. HBase doesn't have this issue since it is HDFS resource based (afaict) and thus is exactly what Hive expects. This is unfortunate since if it wasn't, a lot of the bugs I've encountered in Hive most likely would have been fixed.

Thanks again for the bug report and the follow-up - Grazie!

[chandrakar]

@costin Please help me to resolve the below issues .

i am using hive 0.11

Counters:
Stage-4 is filtered out by condition resolver.
Stage-3 is selected by condition resolver.
Stage-5 is filtered out by condition resolver.
Launching Job 3 out of 3
Number of reduce tasks is set to 0 since there's no reduce operator
java.lang.NullPointerException
at org.apache.hadoop.hive.ql.exec.ExecDriver.addInputPath(ExecDriver.java:843)
at org.apache.hadoop.hive.ql.exec.ExecDriver.addInputPaths(ExecDriver.java:1149)
at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:445)
at org.apache.hadoop.hive.ql.exec.MapRedTask.execute(MapRedTask.java:138)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:144)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)
at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:47)
Job Submission failed with exception 'java.lang.NullPointerException(null)'
java.lang.IllegalArgumentException: Can not create a Path from an empty string
at org.apache.hadoop.fs.Path.checkPathArg(Path.java:127)
at org.apache.hadoop.fs.Path.(Path.java:135)
at org.apache.hadoop.hive.ql.exec.Utilities.getHiveJobID(Utilities.java:418)
at org.apache.hadoop.hive.ql.exec.Utilities.clearMapRedWork(Utilities.java:209)
at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:508)
at org.apache.hadoop.hive.ql.exec.MapRedTask.execute(MapRedTask.java:138)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:144)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)
at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:47)
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MapRedTask
MapReduce Jobs Launched:
Job 0: Map: 1 Cumulative CPU: 6.31 sec HDFS Read: 280 HDFS Write: 0 SUCCESS
Total MapReduce CPU Time Spent: 6 seconds 310 msec

Info : 10:13:29 : 1

You can reach out to me at : yashwant.chandrakar.exp@gmail.com

Thanks and Regards,
Yashwant Chandrakar

[costin]

@chandrakar Please don't highjack an old issue. Use the mailing list or IRC for questions, not the issue tracker.

[chandrakar]

Thanks @costin i will take care of this things from next time .

I will Use the mailing list or IRC for questions.

Thanks

[DeeeFOX]

• Description

  When using hive dealing data, whenever mapred job is submitted, Error of this happen:

  Launching Job 1 out of 3
  Number of reduce tasks is set to 0 since there's no reduce operator
  java.lang.IllegalArgumentException: Can not create a Path from an empty string
      at org.apache.hadoop.fs.Path.checkPathArg(Path.java:126)
      ...
      at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
  Job Submission failed with exception 'java.lang.IllegalArgumentException(Can not create a Path from an empty string)'
  FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask. Can not create a Path from an empty string
  

• Why it happen

  • It is not caused by
    • user prividlege/right of the hadoop/local path of hadoop/hive
    • yarn resourge/node manager problem
    • hadoop/hive/java version
  • It is because

    • file ${HIVE_HOME}/conf/hive-site.sh didn't config right like this:

      ...
      # Folder containing extra ibraries required for hive compilation/execution can be controlled by:
      export HIVE_AUX_JARS_PATH=/home/gzzhaozongfei/soft/mongodb:${HIVE_AUX_JARS_PATH}
      

      The problem is caused by the part: ${HIVE_AUX_JARS_PATH}

• Solution

  • config the ${HIVE_HOME}/conf/hive-site.sh right then it sovled:

    ...
    # Folder containing extra ibraries required for hive compilation/execution can be controlled by:
    export HIVE_AUX_JARS_PATH=/home/gzzhaozongfei/soft/mongodb