| title | description | author | ms.author | ms.service | ms.subservice | ms.topic | ms.date |
|---|---|---|---|---|---|---|---|
Incrementally copy data from a source data store to a destination data store |
These tutorials show you how to incrementally copy data from a source data store to a destination data store. The first one copies data from one table. |
dearandyxu |
yexu |
data-factory |
tutorials |
tutorial |
05/15/2024 |
[!INCLUDEappliesto-adf-asa-md]
In a data integration solution, incrementally (or delta) loading data after an initial full data load is a widely used scenario. The tutorials in this section show you different ways of loading data incrementally by using Azure Data Factory.
In this case, you define a watermark in your source database. A watermark is a column that has the last updated time stamp or an incrementing key. The delta loading solution loads the changed data between an old watermark and a new watermark. The workflow for this approach is depicted in the following diagram:
:::image type="content" source="media/tutorial-incremental-copy-overview/workflow-using-watermark.png" alt-text="Workflow for using a watermark":::
For step-by-step instructions, see the following tutorials:
- Incrementally copy data from one table in Azure SQL Database to Azure Blob storage
- Incrementally copy data from multiple tables in a SQL Server instance to Azure SQL Database
For templates, see the following:
Change Tracking technology is a lightweight solution in SQL Server and Azure SQL Database that provides an efficient change tracking mechanism for applications. It enables an application to easily identify data that was inserted, updated, or deleted.
The workflow for this approach is depicted in the following diagram:
:::image type="content" source="media/tutorial-incremental-copy-overview/workflow-using-change-tracking.png" alt-text="Workflow for using Change Tracking":::
For step-by-step instructions, see the following tutorial:
You can copy the new and changed files only by using LastModifiedDate to the destination store. ADF will scan all the files from the source store, apply the file filter by their LastModifiedDate, and only copy the new and updated file since last time to the destination store. Please be aware that if you let ADF scan huge amounts of files but you only copy a few files to the destination, this will still take a long time because of the file scanning process.
For step-by-step instructions, see the following tutorial:
For templates, see the following:
You can copy new files only, where files or folders has already been time partitioned with timeslice information as part of the file or folder name (for example, /yyyy/mm/dd/file.csv). It is the most performant approach for incrementally loading new files.
For step-by-step instructions, see the following tutorial:
Advance to the following tutorial:
[!div class="nextstepaction"] Incrementally copy data from one table in Azure SQL Database to Azure Blob storage