[SUPPORT] Request for support for Azure Blob Storage Events Source and Events Hoodie Incr Source in DeltaStreamer

[satiricr2d2]

Tips before filing an issue

• Have you gone through our FAQs?

• Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.

• If you have triaged this as a bug, then file an issue directly.

Describe the problem you faced
Since Streaming ingestion using DeltaStreamer from DFS that contains parquet files has a problem as explained in this blog here https://medium.com/apache-hudi-blogs/reliable-ingestion-from-aws-s3-using-hudi-b7d5590c78a9 here, I'm working on implementing a similar setup described in the above mentioned blog for ingesting parquet files stored in Azure blob storage and enable event triggers to Azure storage queue.
Currently, hudi-utilities/sources contains support only for S3 events source (S3EventsSource.java) and incremental pulls from S3 (S3EventsHoodieIncrSource.java). The ingestion pattern doesn't seem to support equivalent Azure cloud stack.

To Reproduce

Expected behavior

Suggested Ingestion Path (Just an example):

  1                                                       2                                                                      3                                               4

Azure Blob Storage -----> Azure Event Grid (for notifications) ------> Azure Storage Queue -----> DeltaStreamer Pull (Hudi Metadata table)

  5                                                         6                                             7                                        8

Hudi Metadata table------> DeltaStreamer Incr Pulls-----> Azure Blob Storage ---> Hudi Source Table.

Support request for reliable ingestion of azure blob storage events from azure storage via azure event grid and azure storage queue to hudi. Reference: https://medium.com/apache-hudi-blogs/reliable-ingestion-from-aws-s3-using-hudi-b7d5590c78a9

Environment Description

• Hudi version : 0.12.0

• Spark version : 3.3

• Hive version :

• Hadoop version :

• Storage (HDFS/S3/GCS..) : Azure Blob Storage / Azure Storage Queue

• Running on Docker? (yes/no) : No

Additional context
Could you please provide any recommendation or ETA on this support request for doing reliable ingestion from Azure Blob Storage/Storage queue using DeltaStreamer ?

Stacktrace

Add the stacktrace of the error.

[satiricr2d2]

@nsivabalan, Could you please provide any timeline on when we can expect this feature in DeltaStreamer?

[xushiyan]

@rtdt99 i've filed this https://issues.apache.org/jira/browse/HUDI-5202

This kind of feature is mostly community-driven so there is no specific timeline. Anyone interested can pick it up and submit a PR for review. cc @kywe665

[yihua]

Hi @rtdt99 Thanks for raising the feature request! This is useful for Azure Blob Storage users. The overall flow looks good. We encourage you to put up the full and detailed design in an RFC (you can follow the RFC process here). The community can help review the RFC and the implementation.

Let's follow up in the JIRA ticket: HUDI-5202. Closing this issue now.