Skip to content

Loading…

hadoop-test-1.0.4.jar fails with local QFS #18

Closed
VladimirStarostenkov opened this Issue · 2 comments

2 participants

@VladimirStarostenkov

Ubuntu 12.04, hadoop 1.0.4, latest QFS, local single test node.

$ hadoop jar /home/cdh/hadoop-1.0.4/hadoop-test-1.0.4.jar mapredtest 100 10000

or any other test fails with

java.io.FileNotFoundException: /user/cdh/mapred.loadtest/genouts: No such file or directory 2
at com.quantcast.qfs.access.KfsAccess.kfs_retToIOException(KfsAccess.java:804)
at com.quantcast.qfs.hadoop.QFSImpl.stat(QFSImpl.java:100)
at com.quantcast.qfs.hadoop.QuantcastFileSystem.listStatus(QuantcastFileSystem.java:122)
at com.quantcast.qfs.hadoop.QuantcastFileSystem.delete(QuantcastFileSystem.java:175)
at org.apache.hadoop.mapred.TestMapRed.launch(TestMapRed.java:550)
at org.apache.hadoop.mapred.TestMapRed.run(TestMapRed.java:825)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
at org.apache.hadoop.mapred.TestMapRed.main(TestMapRed.java:753)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
at org.apache.hadoop.test.AllTestDriver.main(AllTestDriver.java:81)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

$hadoop fs -ls / mkdir / rm etc works fine.

The same tests on the same configuration, but with core-site.xml configured for HDFS, do not fail. I put the body of listStatus into try-catch - tests pass, everything works correctly.

@gofish

Thanks for reporting. I managed to reproduce the error on Ubuntu 12.10 with hadoop 1.0.4 and qfs 1.0.1 as well. I'll check with the core devs and look into the failing test.

@gofish

Looks like QuantcastFileSystem.delete(Path, boolean) violates the documentation for FileSystem.delete() and throws a FileNotFoundException instead of returning false. The with following patch to TestMapRed.java the test succeeds and passes. Will check tomorrow if this behavior is unexpected and should be fixed or can be disabled for stricter compliance.

--- org/apache/hadoop/mapred/TestMapRed.java.orig       2013-02-11 19:51:24.028906319 -0800
+++ org/apache/hadoop/mapred/TestMapRed.java    2013-02-11 19:44:07.242638910 -0800
@@ -547,7 +547,7 @@
     // file of random numbers.
     //
     Path randomOuts = new Path(testdir, "genouts");
-    fs.delete(randomOuts, true);
+    if (fs.exists(randomOuts)) fs.delete(randomOuts, true);


     JobConf genJob = new JobConf(conf, TestMapRed.class);
@@ -595,7 +595,7 @@
     //
     int intermediateReduces = 10;
     Path intermediateOuts = new Path(testdir, "intermediateouts");
-    fs.delete(intermediateOuts, true);
+    if (fs.exists(intermediateOuts)) fs.delete(intermediateOuts, true);
     JobConf checkJob = new JobConf(conf, TestMapRed.class);
     FileInputFormat.setInputPaths(checkJob, randomOuts);
     checkJob.setInputFormat(TextInputFormat.class);
@@ -619,7 +619,7 @@
     // all the files.
     //
     Path finalOuts = new Path(testdir, "finalouts");
-    fs.delete(finalOuts, true);
+    if (fs.exists(finalOuts)) fs.delete(finalOuts, true);
     JobConf mergeJob = new JobConf(conf, TestMapRed.class);
     FileInputFormat.setInputPaths(mergeJob, intermediateOuts);
     mergeJob.setInputFormat(SequenceFileInputFormat.class);
@gofish gofish was assigned
@gofish gofish pushed a commit that closed this issue
Jeremy R. Fishman Return false from QFS.delete() when FileNotFound. Fixes #18. 822c854
@gofish gofish closed this in 822c854
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.