[SUPPORT]Best way to ingest a large number of tables

[ziudu]

Dear,

We're new to Hudi and we would like to know which the best way is to ingest a large number of tables. For example, in the production environment, we have about 70 mysql databases with >1000 tables in total. We'd prefer them all to be ingested with continuous mode in the spirit of data lake.

• option 1, deltastreamer : each table requires a single deltastreamer, so it consumes too much resource. (Is it possible to submit multiple deltastreamer into 1 spark context?)
• option 2, multitabledeltastreamer: it doesn't support MOR yet, which is our preferred format.
• option 3, write our own data ingestion logic with java-client, but it takes some time.

Do you have any suggestions?
Thanks,

[codope]

cc @pratyakshsharma Any particular reason we don't support MOR in HoodieMultiTableDeltaStreamer. What will it take to support it?

cc @n3nash , any better way than option 3 that you could suggest?

[ziudu]

We made a few POC and found:

1. java-client didn't support MOR either so we would not use it.
2. Multitabledeltastreamer did not work correctly in continuous mode. It did work somehow for MOR in single-run mode, but we preferred not to use it as the doc said MOR was not supported. Plus, multitabledeltastreamer runs ingestions in serial mode, not in parallel mode.

For the moment, we will stick to delta streamers, and launch them regularly (every 5-10 minutes) to process change data from Debezium or Golden Gate. Let's see what will happen for 1000 tables.

However, I don't think it's optimized, as

• 1 delta streamer needs at least 1 spark executor, with usually 2GB memory. Most of our tables have only a very small amount of change data (<1MB) during a 5-10 minutes' period. We might need a large Hadoop cluster with enough memory for data ingestion, transformation and PrestoSQL.
• We use spark on Yarn, so it takes 10 seconds to create a Yarn delta streamer application, which we think is not optimized either.

Our final thought:
Is it possible to write a long-running spark application, which listens to multiple data change topics, and writes change data in parallel to Hadoop hoodie tables via pyspark data frame?

[nsivabalan]

@ziudu : I understand you have 1000 tables on the source side. And does each of this map to separate hudi tables? I am trying to understand if you are looking at multi-writer on the hudi end or just ingestion from 1000 source tables to 1000 hudi tables sort of.

[pratyakshsharma]

@codope As part of the first release, it was decided to implement MultiTableDeltaStreamer for only COW and later on extend the support for MOR as well which is not started as of yet. Would you like to take that up?

[ziudu]

@nsivabalan Yes, we have 1000 tables, and each of them map to separate hudi table. 1000 db tables -> 1000 hudi tables.

We're writing a custom spark application with scala, which listens to a number of topics and writes to a number of hudi tables. Something similar to

#2175

The testing result is satisfactory so far and we need to do some performance testing later on. I just want to know if it is the recommended way.

[nsivabalan]

@ziudu : Here is a suggestion. can you get started with Multi-table detlastreamer using COW for now, and we will provide you support. Also, if you plan to build something on your own using java-client, it would be better to invest in adding MOR support to Multi-table deltastreamer and so community at large will benefit. As more and more people from the community starts using it, it will become more and more robust.
Also, I don't see any technical blocker as such to support MOR w/ deltastreamer, just that we did not have much resources.
What do you think?

[nsivabalan]

I tried MOR with multi table deltastreamer and it worked(one time sync) for multiple tables.
May be we can start with MOR table type and you can schedule multi-table deltastreamer once every 5 or 10 mins. Should achieve what you are looking for. In the mean time, we can try to see how to add continuous mode support in multi-table deltastreamer.

[nsivabalan]

I went through the attached code as well #2175 and it looks reasonable. Let us know how it goes.

[nsivabalan]

@ziudu : Is there any update? Anything worked for you, anything that did not work ?

[vinothchandar]

@ziudu this has come up a few times now. Do you mind sharing what you decided to do?