Bulk data.all dataset import

[dlpzx]

Is your feature request related to a problem? Please describe.
I am a customer with a substantial amount of Glue databases and S3 Buckets. I do not want to manually import each dataset into data.all.

Describe the solution you'd like
I want a task to perform a bulk-import of my datasets into data.all. I would provide dataset owner team and initial metadata that applies to multiple datasets at once.

Describe alternatives you've considered

1. Import multiple datasets at once with an ECS task

[dlpzx]

Hi @manjulaK, @chamcca and I have discussed possible ways of implementing this feature. These are some of the options that we considered:

1. First, we thought of having an ECS tasks where you defined the S3 buckets to import and their metadata. Either in batch or one by one with a config file. Defining this kind of config file does not really reduce the time to import datasets, in comparison with importing them from the UI. So we discarded this idea and went for a batch imports of S3 Buckets that share the same metadata (Environment, AdminGroup, Confidentiality, Topics and Tags). The ECS task would take as inputs a list of buckets and this metadata.
2. Then we considered how would the ECS task look like. Whenever we import a dataset 3 things happen in the backend: new RDS row for the dataset, new OpenSearch index (both by graphQL Lambda) and deployment of dataset stack by cdkproxy ECS task. This happens for each imported dataset and we want to keep atomicity between imported datasets. Let's say that one S3 bucket of the defined list does not exist, we need to isolate its failure to only the wrong import.
   For this reason we would prefer to has separated tasks for each imported dataset, which would need to be handled by an SQS queue to avoid limitations on the number of tasks being run at a single moment.
3. we realized that by calling the API import we are queuing the tasks, do ideally we would call this API multiple times. At the end the simplest most effective solution is to provide a UI API to perform a "batch import"

[noah-paige]

Closing this issue in favor of the Data.all SDK and CLI initiative

Making SDK and CLI tools available for data.all should allow for automation, scripting, and integration into existing workflows, such as automating the import of multiple datasets via a configuration file