Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

vdk-core: support multiple ingesters from different types to be used with job_input #3293

Closed
duyguHsnHsn opened this issue Apr 4, 2024 · 0 comments

Comments

@duyguHsnHsn
Copy link
Collaborator

Currently, send_tabular_data_for_ingestion and send_object_for_ingestion in job_input only work with the default ingestion database that is set, instead of this we should support them to work with all the registered databases in


Their interfaces should look like this:

def run(job_input):
         .......
        job_input.send_object_for_ingestion(
                         payload=dict(id="Hello World!"), 
                         destination_table="hello_world", 
                         database="TRINO" 
                         )
         job_input.send_object_for_ingestion(
                          payload=dict(id="Hello World!"), 
                          destination_table="hello_world",
                          database="IMPALA"                        
                         )

For now, for each database type we can only have single ingester registered. Multiple connections from the same time will be implemented after #3222

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant