Skip to content
Metadata driven processing framework for Azure Data Factory using Azure SQLDB and Azure Functions
TSQL PowerShell C#
Branch: master
Clone or download

Latest commit

Fetching latest commit…
Cannot retrieve the latest commit at this time.

Files

Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
ARM Templates/Data Factory
DataFactory
DeploymentTools
Images
MetadataDB Final pre release checks and tweaks. Mar 24, 2020
PipelineExecutor
.gitattributes
.gitignore
ProcessingFramework.sln
README.md

README.md

ADF.procfwk

alt text

Code Project Overview

This open source code project delivers a simple metadata driven processing framework for Azure Data Factory (ADF). The framework is delivered by coupling ADF with an Azure SQL Database that houses execution stage and pipeline information that is later called using an Azure Functions App. The parent/child metadata structure firstly allows stages of dependencies to be executed in sequence. Then secondly, all pipelines within a stage to be executed in parallel offering scaled out control flows where no inter-dependencies exist for a given stage.

The framework is designed to integrate with any existing Data Factory solution by making the lowest level executor a stand alone processing pipeline that is wrapped in a higher level of controlled (sequential) dependencies. This level of abstraction means operationally nothing about the monitoring of orchestration processes is hidden in multiple levels of dynamic activity calls. Instead, everything from the processing pipeline doing the work can be inspected using out-of-the-box ADF features.

This framework can also be used in any Azure Tenant and allow the creation of complex control flows across multiple Data Factory resources by connecting Service Principal details to targeted Subscriptions > Resource Groups > Data Factory's and Pipelines, this offers very granular administration over data processing components in a given environment.

Authors

Who Details
Paul Andrew @mrpaulandrew
paul@mrpaulandrew.com
https://mrpaulandrew.tech

Development Backlog

Go to GitHub Kanban board...

Resources and Content

Blogs mrpaulandrew.com/ADF.procfwk
GitHub github.com/mrpaulandrew/ADF.procfwk
Twitter #ADFprocfwk

Release Details

Version Overview Related Blog(s)
1.2 Execution Restartability, plus:
  • Data Factory annotations and descriptions.
  • Database covering indexes.
  • Pipeline log status changed from 'Started' to 'Preparing'.
  • Pipeline log start date/time now set in child pipeline.
ADF.procfwk v1.2 - Execution Restartability
1.1 Service Principal Handling via Metadata, plus:
  • Data Factory table.
  • Properties table and view.
  • Function body bug fix.
  • New sample data.
ADF.procfwk v1.1 - Service Principal Handling via Metadata
1.0 Simple framework designed and base compontents built.

  • Part 1 - Design, concepts, service coupling, caveats, problems.
  • Part 2 - Database build and metadata.
  • Part 3 - Data Factory build.
  • Part 4 - Execution, conclusions, enhancements.
Creating a Simple Staged Metadata Driven Processing Framework for Azure Data Factory Pipelines
You can’t perform that action at this time.