For Extract & Load ( EL ) I am using Airbyte. What is several option for Dagster ?

[SaraNofa]

Hello,

So far I am using Airbyte for Extract & Load ( EL ). I am thinking several option especially - Write custom code using Python.

It is possible I write custom code for EL in Python and Dagster can call this custom code ?

Need guidance

[apoorva-01]

Yeah, that's basically Dagster's whole model. An asset is just a Python function, so your custom EL code goes right inside one and Dagster handles the scheduling, retries, and lineage around it:

import dagster as dg

@dg.asset
def my_table():
    rows = extract_from_source()   # your code
    load_into_warehouse(rows)      # your code

That's the most flexible path and it's fine to run with it.

If you'd rather not hand-roll every connector, there are two first-party EL integrations worth knowing:

• dagster-dlt (dltHub) is the natural fit coming from Airbyte. You define the source in Python and it turns dlt resources into Dagster assets. Best for APIs and messy/unstructured sources.
• dagster-sling is great for straight DB-to-DB / warehouse / file replication driven by a YAML config.

(Heads up: the old dagster-embedded-elt package is deprecated now, those two replaced it.)

And if you want to keep Airbyte for some sources, dagster-airbyte lets Dagster trigger your existing connections, so it doesn't have to be all-or-nothing.