text output and combiner don't work together

[piccolbo]

It was reported by Saar on the RHadoop Google group https://groups.google.com/d/topic/rhadoop/5yYKZZLSX8U/discussion. It seems a consequence of mapper and reducer writing out in two different formats, and the reducer expecting the one from the mapper. When the combiner is activated, it outputs the reducer output format which the reducer then can't read. This warning is telling:

52 WARN streaming.PipeMapRed: java.io.IOException: wrong key class: class org.apache.hadoop.io.Text is not class org.apache.hadoop.typedbytes.TypedBytesWritable

Investigating.

[piccolbo]

Repro with

from.dfs(mapreduce(to.dfs(1:10), combine = T, reduce = function(k,vv) keyval(NULL,sum(unlist(vv))), output.format="csv"), format="csv")

[piccolbo]

Entered a comment here: https://issues.apache.org/jira/browse/HADOOP-1722?focusedCommentId=13404201#comment-13404201 as this is where binary formats for hadoop streaming were introduced and I suspect they did not foresee the use of streaming combiners, added way later with HADOOP-4842. I added a comment there too. I am trying to understand what the intent was when both binary formats and streaming combiners were added to hadoop.

[piccolbo]

Updated test case to

from.dfs(mapreduce(to.dfs(1:10), combine = T, map = function(k,v) keyval(1,v),  reduce = function(k,vv) keyval(1,sum(unlist(vv))), output.format="csv"), format="csv")

[piccolbo]

this is now RevolutionAnalytics/rmr2#16