Implement HoodieRealTimeInputFormat

[vinothchandar]

This houses the merge-on-read record reader

[vinothchandar]

Think about supporting multiple delta files - on compaction failures
Fall back of on-disk merge if the delta file size exceeds certain limit

[vinothchandar]

Agree on approach in https://gist.github.com/prazanna/698459049447d8898a9de11e3863e99d , wrapping is the natural way to go.

But planning to approach the reading a little differently..

1. next() on underlying recordReader to get new array writable (need to add projection to get _hoodie_record_key out always)
2. Somehow turn the original avro delta logs into a HashMap<_hoodie_record_key, ArrayWritable> (this needs some tricky maneuvers)
3. Then the logic is pretty simple.. where we return either from 1 or 2

[prazanna]

Pasting a scrap of thoughts/impl of the record reader - https://gist.github.com/prazanna/698459049447d8898a9de11e3863e99d

@vinothchandar

[prazanna]

Think about supporting multiple delta files - on compaction failures
Fall back of on-disk merge if the delta file size exceeds certain limit

Just a note - I guess we are okay to assume that we are able to merge in all the updates in memory with HoodieAvroReader. #134 tracks the disk spill if needed.

[vinothchandar]

Upon more digging, it all boils down to the following to get the merging right

• Projecting only requested fields in query off Avro records

Reading using the a sub schema built off conf.get(ColumnProjectionUtils.READ_COLUMN_NAMES_CONF_STR) should do the trick. Need to test with nested records and complex data types. Then compare to Parquet projections, in terms of order & so forth and make sure its all good.

http://grepcode.com/file/repo1.maven.org/maven2/org.apache.avro/avro/1.7.7/org/apache/avro/generic/GenericDatumReader.java#GenericDatumReader.read%28java.lang.Object%2Corg.apache.avro.io.Decoder%29 is how the data is read in AvroRecordReader (link)

• Converting the GenericRecord to same ArrayWritable

We need to get the types here and turn it into ArrayWritable

• Extra projection added for _hoodie_record_key should be removed out in the end

Should be zero effect to the change above

[vinothchandar]

As for the design of the InputFormat itself, Hive seems to extend FileSplit across the board

https://github.com/apache/hive/search?p=3&q=FileSplit&type=&utf8=✓

ORC Split : https://github.com/apache/hive/blob/ff67cdda1c538dc65087878eeba3e165cf3230f4/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcNewSplit.java
HBase Split : https://github.com/apache/hive/blob/ff67cdda1c538dc65087878eeba3e165cf3230f4/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSplit.java

Of the two places I saw hardcoding of checks, all of them check for instanceof, so a subclass should be fine.

https://github.com/apache/hive/blob/a1cbccb8dad1824f978205a1e93ec01e87ed8ed5/ql/src/java/org/apache/hadoop/hive/ql/io/parquet/ParquetRecordReaderBase.java#L82
https://github.com/apache/hive/blob/ff67cdda1c538dc65087878eeba3e165cf3230f4/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HiveSplitGenerator.java#L305