Beam IO module to support incremental tailing of Hoodie Hive/Spark tables

[vinothchandar]

No description provided.

[vectorijk]

I am from Beam Community. Love to see if there is any process on this issue. Also, I would love to give some my help on this one.

[vinothchandar]

Hi @vectorijk, the idea here was to see if we can

• Build a HoodieIO - that can support streaming semantics on data in hdfs/s3/gcs in a standard way. There is nt one now IIAC
• Potentially use Hoodie to manage intermediate state to publish corrections.

Then, we could support three style of pipelines : Streaming, Incremental, Batch - to make the spectrum of pipeline latencies more continuous . More here. https://www.oreilly.com/ideas/ubers-case-for-incremental-processing-on-hadoop

That said, this is not being actively prioritized. We are mostly focussing on the next generation of hoodie storage/async compactions/usability cleanups and such..

Love to take contributions! we can add a hoodie-beam module. :)

[vinothchandar]

moved to https://issues.apache.org/jira/browse/HUDI-60