Permalink
Browse files

Explicitly disable map speculative execution, since LzoIndexOutputFor…

…mat doesn't work with it.

This could be fixed, but would require writing a proper OutputCommiter. Patches welcome.

Bumps version to 0.4.2
  • Loading branch information...
1 parent 98c763d commit 1053d83c2e59f90a16a35c040ec8711d8a5a649f @toddlipcon toddlipcon committed Apr 26, 2010
Showing with 6 additions and 1 deletion.
  1. +1 −1 build.xml
  2. +5 −0 src/java/com/hadoop/compression/lzo/DistributedLzoIndexer.java
View
@@ -28,7 +28,7 @@
<property name="Name" value="Hadoop GPL Compression"/>
<property name="name" value="hadoop-lzo"/>
- <property name="version" value="0.4.1"/>
+ <property name="version" value="0.4.2"/>
<property name="final.name" value="${name}-${version}"/>
<property name="year" value="2008"/>
@@ -93,6 +93,11 @@ public int run(String[] args) throws Exception {
job.setOutputKeyClass(Path.class);
job.setOutputValueClass(LongWritable.class);
+ // The LzoIndexOutputFormat doesn't currently work with speculative execution.
+ // Patches welcome.
+ job.getConfiguration().setBoolean(
+ "mapred.map.tasks.speculative.execution", false);
+
job.setJarByClass(DistributedLzoIndexer.class);
job.setInputFormatClass(LzoSplitInputFormat.class);
job.setOutputFormatClass(LzoIndexOutputFormat.class);

0 comments on commit 1053d83

Please sign in to comment.