You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
In order to support ingestion into multiple databases, we should be able to easily instantiate multiple ingestion objects and register them under a name. We should also be able to tell the ingestion API which named ingester to use.
To achieve this, we need
A way to instantiate ingesters from simple dict configs
A way to register ingester classes with vdk-core
A built-in plugin that initializes all the named ingesters based on the available configurations (whatever they may be, the actual configurations are out of scope) and stores them in-memory.
Pass an ingester name to job_input.send_object_for_ingestion(). Note that this might become a separate ticket due to multi-threading when doing ingestion.
Note that this should work for non-database ingesters as well, e.g. ingesters that use an API, local files or other methods.
classIIngesterPlugin()
@classmethod@abstractmethoddef_from_dict(cls, **kwargs):
""" override this if you want to support multiple connections and ingestion operations """pass
ingest_to_oracle.py
classIngestToOracle(IIngesterPlugin):
def__init__(self, name: str, connections: ManagedConnectionRouter):
self.conn: PEP249Connection=connections.open_named_connection(name).connect()
self.cursor: ManagedCursor=self.conn.cursor()
@classmethoddef_from_dict(cls, **kwargs):
# config and intialize from dict
Creates all the named ingester methods based on all the classes that were previously registered. For now, it can just do nothing until we add a configuration API.
Overview
In order to support ingestion into multiple databases, we should be able to easily instantiate multiple ingestion objects and register them under a name. We should also be able to tell the ingestion API which named ingester to use.
To achieve this, we need
Note that this should work for non-database ingesters as well, e.g. ingesters that use an API, local files or other methods.
Pseudocode
ingester_router.py
ingester_plugin.py
ingest_to_oracle.py
oracle_plugin.py
ingester_config_plugin.py
Creates all the named ingester methods based on all the classes that were previously registered. For now, it can just do nothing until we add a configuration API.
Acceptance criteria
The text was updated successfully, but these errors were encountered: