[HUDI-4142][RFC-64] New APIs to facilitate faster Query Engine integrations - #7080
[HUDI-4142][RFC-64] New APIs to facilitate faster Query Engine integrations#7080alexeykudinkin wants to merge 14 commits into
Conversation
xushiyan
added
priority:blocker
2df0c0f to
03354f8
Compare
a532dbe to
10fc1a9
Compare
10fc1a9 to
a118158
Compare
alexeykudinkin
changed the title
alexeykudinkin
changed the title
| ``` | ||
|
|
||
| **Expression** (**Predicate**) | ||
|
|
There was a problem hiding this comment.
only use for dataSkipping/partition prune ？
we donnot think we should use those predicate for engine scan action。
There was a problem hiding this comment.
Correct this will only be used for data filtering (skipping, partition-pruning, filters push-down)
| List<PartitionSnapshot> listFilesAt(HoodieInstant instant, Predicate[] filters); | ||
| // Lists files added visible/reachable at the instant `to`, that were | ||
| // added no earlier than at the instant `from` | ||
| List<PartitionIncrementalSnapshot> listFilesBetween(HoodieInstant from, HoodieInstant to, Predicate[] filters); |
There was a problem hiding this comment.
will FileIndex cache all file information to memory just like BaseHoodieTableFileIndex？
if yes， I think it's not friendly to presto
| aforementioned log scanner API. Then, in Presto we can create `RecordCursor` | ||
| wrapping around `RecordReader`. The key advantage here is that | ||
| the `HoodieRealtimeRecordReader` won't have to spend cycles | ||
| converting `ArrayWritable` to `Avro` and vice-versa as it does today. |
There was a problem hiding this comment.
Currently for hive/presto, we use MapredParquetInputFormat to read the parquet file and return ArrayWritable records.
Do we want to implement our own Parquet reading logic and return avro type directly to avoid spend cycles
converting ArrayWritable to Avro？
There was a problem hiding this comment.
@xiarixiaoyao Good question! We don't intend to implement our own Parquet reading logic. The work here will build on top of RFC-46. So, in this case, we will have an implementation of HoodieRecord and HoodieRecordMerger( as proposed in RFC-46) for Hive ArrayWritable.
There was a problem hiding this comment.
Will custom HoodieRecordMerger implementations be supported and if so how will the reader load those custom implementations that are not part of open source?
There was a problem hiding this comment.
@the-other-tim-brown this is covered in RFC-46: If you want to implement custom merging logic, then
- When creating your table you specify unique MERGER_STRATEGY uuid (identifying your merging strategy/semantic)
- During writing you specify MERGER_IMPLS, which are actual engine-specific RecordMerger impls, that realize particular merging strategy (set as MERGER_STRATEGY)
- When writing Hudi will take up MERGER_IMPLS, filter the ones based on the MERGER_STRATEGY uuid, and use them for merging (based on which engine it's running on)
|
|
||
| ## Appendix A: Hive-specific HoodieRecord implementation | ||
|
|
||
| Here `HoodieRecord` is an abstraction as defined in RFC-46 and it will provide |
There was a problem hiding this comment.
i can help implement it.
Add more details about user-friendly APIs Add expression API open question
Extracted HoodieHiveRecord impl section to appendix
a118158 to
3923bcd
Compare
codope
changed the title
6 participants
Change Logs
This RFC proposes a new set of higher-level Table Spec APIs that would allow us to up-level our current integration model with new Query Engines, enabling faster turnaround for such integrations.
Impact
RFC for discussion
Risk level (write none, low medium or high below)
none
Documentation Update
Implementation of this RFC would require considerable documentation updates to reflect the newly proposed approach of integration for Query Engines as well as user-advised higher-level APIs.
Contributor's checklist