12 Metadata‐Driven ELT Pattern for Mirrored Databases

Fabric Accelerator and Mirrored Databases

Ingest once, reuse everywhere. Build data products directly on OneLake without creating another copy of the data.

Microsoft Fabric Mirroring provides low-cost, low-latency replication of operational data into OneLake, enabling near real-time analytics without complex ingestion pipelines. As the catalog of supported source systems continues to grow, data practitioners can increasingly adopt a OneLake-native approach, treating mirrored data as the bronze layer and focusing engineering effort on creating trusted data products.

Fabric Accelerator supports a new OneLake-Native ELT Pattern for Microsoft Fabric Mirrored Databases. Because mirrored data already lands in OneLake, no separate ingestion process is required. Instead, the Fabric Accelerator metadata framework has been enhanced to orchestrate and automate transformations through the silver and gold layers while maintaining the same governance, quality, and engineering rigor as traditional ingestion-based pipelines.

Aligned with Fabric's One Copy architecture, this pattern enables organizations to build trusted data products directly from shared OneLake assets without re-ingesting or duplicating data. While initially focused on Mirrored Databases, the same approach can be applied to other OneLake-native sources, including Fabric SQL Databases, Lakehouses, Warehouses, Data Mesh domains, and OneLake Shortcuts managed by teams across the organization.

Key Benefits

• Eliminates unnecessary ingestion for OneLake-native sources
• Reduces data duplication and movement
• Accelerates delivery of silver and gold data products
• Enables reuse of shared enterprise data assets
• Extends Fabric's One Copy principle across data engineering workloads
• Provides a consistent metadata-driven approach regardless of data origin

Implementing Fabric Accelerator with Mirrored Databases

This implementation showcases the OneLake-Native ELT Pattern using Microsoft Fabric Mirroring. Mirrored data lands directly in OneLake and serves as the bronze layer, allowing Fabric Accelerator to focus on metadata-driven transformations into silver and gold data products.

1. Deploy the Mirroring Source

This accelerator uses the Wide World Importers Azure SQL Database as the source for Fabric Mirroring.

• Deploy Wide World Importers in your Azure subscription.
• Ensure the database meets the Fabric Mirroring prerequisites, including compute tier requirements.
• Complete the Fabric Accelerator deployment by following the set-up guide

2. Deployed Fabric Assets

Mirrored Database

The deployment creates a mirrored database named:

• WideWorldImporters-mirror

Mirroring is automatically initialized and configured for the following tables:

Schema	Tables
Application	PaymentMethods, People, TransactionTypes
Purchasing	PurchaseOrders, SupplierTransactions
Sales	CustomerTransactions, Orders

Fabric Connection

A Fabric connection named the following is created:

<WORKSPACE_NAME>-WWI-mirror

ELT Pipelines

The following pipelines are deployed:

• Master ELT MirrorDB
• Ingest MirrorDB Table

These pipelines do not ingest data. They manage ELT metadata required to orchestrate transformations on data already mirrored into OneLake.

For more information, see MirrorDB ELT Pipelines.

ELT Metadata

The setup automatically creates metadata entries in:

• ELT.IngestDefinition
• ELT.L1TransformDefinition
• ELT.L2TransformDefinition

Filter records using:

SourceSystemName = 'WWI-mirror'

Before running the framework, update the ComputeName column in ELT.L1TransformDefinition with the Item ID of your L1Transform-Generic-Fabric notebook.

Gold Data Warehouse

Sample stored procedures are deployed with the prefix:

mirror_

---

3. Validate Mirroring

Allow several minutes for data replication to complete.

If mirroring fails to start:

1. Review the Fabric Mirroring Troubleshooting Guide.
2. Check the known CI/CD deployment issue documented under T-SQL Queries for Troubleshooting.
3. Manually grant Read and Write permissions to the Azure SQL Managed Identity if required.

https://github.com/bennyaustin/fabric-accelerator/blob/main/images/mirroring/Manage%20Permissions.png

---

4. Run the ELT Framework

Execute the pipelines in the following order:

1. Master ELT MirrorDB
2. Master Level1 Transform (Bronze → Silver)
3. Master Level2 Transform (Silver → Gold)

5. Validate Results

Confirm that:

• Mirrored source data is available in OneLake.
• Silver Delta tables have been populated.
• ELT control tables contain successful execution records.
• Gold snapshot tables have been generated.
• Pipeline and notebook executions are visible in the Fabric Observability experience.

---

Architecture Summary

With Fabric Mirroring, source data is replicated directly into OneLake using low-latency, managed replication. Fabric Accelerator treats mirrored data as the bronze layer and uses metadata-driven ELT patterns to create silver and gold data products without re-ingesting or duplicating data, aligning with Fabric's One Copy architecture.