Skip to content

Full load versus Incremental

Jump to bottom
Erwin de Kreuk edited this page Apr 28, 2026 · 3 revisions

The Fabric Metadata‑Driven (FMD) Framework supports both full and incremental data loading. The loading behavior is centrally controlled via metadata in the Data Landingzone Layer and applied consistently across Bronze and Silver layers.

Data Landingzone Layer

Incremental loading is configured on entity level using the LandingzoneEntity table.

Required Columns

To enable incremental loading, the following columns must be defined:

Column Data Type Constraints Description
IsIncremental bit NOT NULL, DEFAULT ((0)) Indicates if incremental loading is enabled
IsIncrementalColumn nvarchar(50) NULL Column used for incremental loading

Configuration Rules

  • When IsIncremental = 1:

    • IsIncrementalColumn must be provided.

  • When IsIncremental = 0:

    • The entity is processed as a full load.

First Incremental Load Behavior

When an entity is loaded incrementally for the first time:

  1. A full load is executed automatically.
  2. After the load completes, a record is inserted into:
[execution].[LandingzoneEntityLastLoadValue]

This table stores the last processed value of the incremental column and is used as the starting point for subsequent incremental loads.

Forcing a Full Reload

If you want to reload an entity fully after it has been processed incrementally, you have two options:

Option 1 – Remove the Last Load Value

Delete the corresponding row from:

[execution].[LandingzoneEntityLastLoadValue]

Option 2 – Reset the Load Value

Update the stored value to an earlier date or value, forcing the next run to reload a larger dataset.

✅ Note: No changes to the LandingzoneEntity configuration are required.

Bronze Layer Behavior

The behavior of the Bronze layer depends on whether incremental loading is enabled.

Incremental Enabled (IsIncremental = 1)

  • All incoming records are appended to the Bronze table.
  • No deletions are applied.
  • Historical changes are preserved.

Incremental Disabled (IsIncremental = 0)

  • A full compare is performed between:

    • Data Landingzone (current snapshot)
    • Existing Bronze records

  • Records that no longer exist in the source are:

    • Physically deleted from the Bronze layer

This allows the framework to detect and propagate source-system deletions.

Silver Layer Behavior

The Silver layer applies consistent business rules, regardless of load type.

  • No direct changes are required to Silver logic when switching between full and incremental loads.
  • When records are deleted in the Bronze layer:
    • They are soft-deleted in Silver
    • Typically implemented via:
      • IsDeleted flag
      • DeletedDate timestamp

This ensures:

  • Full historical traceability
  • No physical deletes at the curated layer

Clone this wiki locally