Home

Welcome to the CDC wiki!

Compared to ETL

What it does:

• CDC is all about capturing and tracking changes made to the source data. Instead of moving the entire dataset, it identifies what’s been added, updated, or deleted since the last extraction.
• ETL is a comprehensive process that involves Extracting data from the source, Transforming it according to predefined rules, and Loading it into the target destination.

How it works:

• CDC: typically two approaches
  • log-based CDC: Log-based captures changes directly from the database transaction logs
  • time-based CDC: time-based periodically checks for changes based on a predefined schedule
• ETL: It pulls the entire dataset, performs transformations (cleaning, aggregating, etc.), and then loads the transformed data into the target system.

Use cases:

• CDC: Ideal for scenarios where you need real-time or near-real-time updates, like in data warehouses or systems where staying current is crucial.
• CDC: CDC can be more cost-effective due to its focus on changes, saving on resources and time.
• ETL: Perfect for scenarios where data needs substantial transformation before reaching its final destination, like data warehousing, business intelligence, or historical data analysis.