hadoop-compat addons project with HadoopDataSource

[rmetzger]

I picked up the code of #424 and enhanced it with the following:

• make hadoop-compat compatible to hadoop yarn and remove code in package org.apache.hadoop.
• Extended usercode object wrapper (@aljoscha, please validate my changes), fixed InputFormat serialization
• introduce pluggable type converter
• Add a generic wrapper for Hadoop's Writable and WritableComparable.

The generic wrapper allows to do the following: (The example is from Mongodb's Hadoop InputFormat)

public void map(Record record, Collector<Record> out) throws Exception {
    WritableWrapper wrap = record.getField(0, WritableWrapper.class);
    Writable wr = wrap.value();
    Writable valWr = record.getField(1, WritableWrapper.class).value();
    BSONWritable bson = (BSONWritable) wr;
    BSONWritable value = (BSONWritable) valWr;
    System.err.println("bson value has "+value.toString());
}

Lets discuss if we want the project being called "hadoop-compat" or if you prefer "hadoop-compatability" ?

Open issues:

• If the Hadoop IF is a FileInputFormat, we should make the split assignment file locality aware
• Add junit tests for various formats such as ORC, Parquet and Avro.
• Write website documentation for this code.
• Add HadoopDataSink
• See if it is possible to map hadoop counters to our accumulators

[uce]

Nice. I vote for hadoop-compatability as compat is imho not a standard acronym.

What do you mean with "Add HadoopDataSource"? Typo for sink?

[rmetzger]

I saw compat being used by some Linux developers ;)
And the source was a typo, I meant sink.

I added an additional converter that allows to use any Hadoop Writable!

[fhueske]

Not sure, if I would call it either.
This feature gives "only" support for Hadoop InputFormats, but there is lots of other stuff in Hadoop which we do not support right now such as OutputFormats, Map, and Reduce functions.
If we plan to extend the support for these interfaces as well, I would go with hadoop-compatibility.
Otherwise, hadoop-input would be a good name IMHO.

[rmetzger]

The plan is to put everything related to hadoop compatability into this package.
I think the next step would be a OutputFormat.

I will change the name!

[StephanEwen]

We are thinking about adding proper interface compatibility with Hadoop
later. That could all be in one project, called hadoop-compatibility.

On Mon, Jan 27, 2014 at 2:34 AM, Robert Metzger notifications@github.comwrote:

The plan is to put everything related to hadoop compatability into this
package.
I think the next step would be a OutputFormat.

I will change the name!

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/437#issuecomment-33356572
.

[rmetzger]

@aljoscha: Could you review this change? (I committed it already to master) ... but it would be good if you can confirm my approach.