Merge pull request #14878 from jmart1428/edit-orchestration

Pipeline:[Canopy] Updated 'Choose a data pipeline orchestration technology in Azure

Author: JillGrant615

docs/data-guide/technology-choices/pipeline-orchestration-data-movement.md

@@ -1,5 +1,5 @@
 ---
-title: Choose a data pipeline orchestration technology
+title: Choose a Data Pipeline Orchestration Technology
 description: Choose an Azure data pipeline orchestration technology to automate pipeline orchestration, control flow, and data movement workflows.
 author: claytonsiemens77
 ms.author: pnp
@@ -12,81 +12,91 @@ ms.subservice: architecture-guide
 
 # Choose a data pipeline orchestration technology in Azure
 
-Most big data solutions consist of repeated data processing operations, encapsulated in workflows. A pipeline orchestrator is a tool that helps to automate these workflows. An orchestrator can schedule jobs, execute workflows, and coordinate dependencies among tasks.
+Most big data solutions consist of repeated data processing operations, encapsulated in workflows. A pipeline orchestrator helps automate these workflows. It can schedule jobs, run workflows, and coordinate dependencies among tasks.
 
-## What are your options for data pipeline orchestration?
+## Options for data pipeline orchestration
 
-In Azure, the following services and tools will meet the core requirements for pipeline orchestration, control flow, and data movement:
+In Azure, the following services and tools meet the core requirements for pipeline orchestration, control flow, and data movement:
 
-- [Azure Data Factory](/azure/data-factory/)
-- [Oozie on HDInsight](/azure/hdinsight/hdinsight-use-oozie-linux-mac)
+- [Azure Data Factory](/azure/data-factory)
+- [Apache Oozie on Azure HDInsight](/azure/hdinsight/hdinsight-use-oozie-linux-mac)
 - [SQL Server Integration Services (SSIS)](/sql/integration-services/sql-server-integration-services)
+- [Fabric Data Factory](/fabric/data-factory/data-factory-overview)
 
-These services and tools can be used independently from one another, or used together to create a hybrid solution. For example, the Integration Runtime (IR) in Azure Data Factory V2 can natively execute SSIS packages in a managed Azure compute environment. While there is some overlap in functionality between these services, there are a few key differences.
+You can use these services and tools independently or combine them to create a hybrid solution. For example, the integration runtime (IR) in Data Factory V2 can natively run SSIS packages in a managed Azure compute environment. These services share some functionality, but they have a few key differences.
 
-## Key Selection Criteria
+## Key selection criteria
 
-To narrow the choices, start by answering these questions:
+To narrow your options, consider the following factors:
 
-- Do you need big data capabilities for moving and transforming your data? Usually this means multi-gigabytes to terabytes of data. If yes, then narrow your options to those that best suited for big data.
+- Determine whether you need big data capabilities to move and transform your data. These capabilities typically use multiple gigabytes (GBs) to terabytes (TBs) of data. If you require these capabilities, choose a service designed for big data.
 
-- Do you require a managed service that can operate at scale? If yes, select one of the cloud-based services that aren't limited by your local processing power.
+- Identify whether you need a managed service that can operate at scale. If you do, choose a cloud-based service that doesn't depend on your local processing power.
 
-- Are some of your data sources located on-premises? If yes, look for options that can work with both cloud and on-premises data sources or destinations.
+- Check whether you have data sources located on-premises. If you do, choose a service that supports both cloud and on-premises data sources or destinations.
 
-- Is your source data stored in Blob storage on an HDFS filesystem? If so, choose an option that supports Hive queries.
+- Check whether you store source data in blob storage on a Hadoop Distributed File System (HDFS). If you do, choose a service that supports Hive queries.
+
+- Determine whether you need advanced orchestration for complex extract, transform, and load (ETL) workflows across multiple data sources. If you do, choose Fabric Data Factory because it provides a set of connectors, pipeline orchestration, and integration with both on-premises and cloud environments. It's ideal for enterprise-scale data movement and transformation.
 
 ## Capability matrix
 
 The following tables summarize the key differences in capabilities.
 
 ### General capabilities
 
-| Capability | Azure Data Factory | SQL Server Integration Services (SSIS) | Oozie on HDInsight
-| --- | --- | --- | --- |
-| Managed | Yes | No | Yes |
-| Cloud-based | Yes | No (local) | Yes |
-| Prerequisite | Azure Subscription | SQL Server  | Azure Subscription, HDInsight cluster |
-| Management tools | Azure Portal, PowerShell, CLI, .NET SDK | SSMS, PowerShell | Bash shell, Oozie REST API, Oozie web UI |
-| Pricing | Pay per usage | Licensing / pay for features | No additional charge on top of running the HDInsight cluster |
+| Capability | Data Factory | SSIS | Oozie on HDInsight | Fabric Data Factory |
+| --- | --- | --- | --- | --- |
+| Managed | Yes | No | Yes | Yes |
+| Cloud-based | Yes | No (local) | Yes | Yes |
+| Prerequisite | Azure subscription | SQL Server  | Azure subscription, HDInsight cluster | Fabric-enabled workspace |
+| Management tools | Azure portal, PowerShell, CLI, .NET SDK | SQL Server Management Studio (SSMS), PowerShell | Bash shell, Oozie REST API, Oozie web user interface (UI) | Copy job, mirroring, pipeline activities, Dataflow Gen2 |
+| Pricing | Pay per usage | Licensing, extra features add cost | Included with HDInsight cluster | Included with Fabric capacity |
 
 ### Pipeline capabilities
 
-| Capability | Azure Data Factory | SQL Server Integration Services (SSIS) | Oozie on HDInsight
-| --- | --- | --- | --- |
-| Copy data | Yes | Yes | Yes |
-| Custom transformations | Yes | Yes | Yes (MapReduce, Pig, and Hive jobs) |
-| Azure Machine Learning scoring | Yes | Yes (with scripting) | No |
-| HDInsight On-Demand | Yes | No | No |
-| Azure Batch | Yes | No | No |
-| Pig, Hive, MapReduce | Yes | No | Yes |
-| Spark | Yes | No | No |
-| Execute SSIS Package | Yes | Yes | No |
-| Control flow | Yes | Yes | Yes |
-| Access on-premises data | Yes | Yes | No |
+| Capability | Data Factory | SSIS | Oozie on HDInsight | Fabric Data Factory |
+| --- | --- | --- | --- | --- |
+| Copy data | Yes | Yes | Yes | Yes |
+| Custom transformations | Yes | Yes | Yes (MapReduce, Pig, and Hive jobs) | Yes |
+| Azure Machine Learning scoring | Yes | Yes (with scripting) | No | Yes (via integration) |
+| HDInsight on-demand | Yes | No | No | No |
+| Azure Batch | Yes | No | No | Yes |
+| Pig, Hive, and MapReduce | Yes | No | Yes | Yes |
+| Apache Spark | Yes | No | No | Yes |
+| Run SSIS packages | Yes | Yes | No | Yes |
+| Control flow | Yes | Yes | Yes | Yes |
+| Access on-premises data | Yes | Yes | No | Yes |
 
 ### Scalability capabilities
 
-| Capability | Azure Data Factory | SQL Server Integration Services (SSIS) | Oozie on HDInsight
-| --- | --- | --- | --- |
-| Scale up | Yes | No | No |
-| Scale out | Yes | No | Yes (by adding worker nodes to cluster) |
-| Optimized for big data | Yes | No | Yes |
+| Capability | Data Factory | SSIS | Oozie on HDInsight | Fabric Data Factory |
+| --- | --- | --- | --- | --- |
+| Scale up | Yes | No | No | Yes |
+| Scale out | Yes | No | Yes (by adding worker nodes to cluster) | Yes |
+| Optimized for big data | Yes | No | Yes | Yes |
+
+## Alternative approach
+
+In addition to traditional batch-based orchestration, your platform can also use real-time intelligence through the [Fabric Real-Time Intelligence feature](/fabric/real-time-intelligence/event-streams/create-manage-an-eventstream). This approach enables continuous streaming data ingestion, in-flight transformation, and event-driven workflows so that you can respond instantly as data arrives. It supports high-value scenarios such as Internet of Things (IoT) telemetry processing, fraud detection, and operational monitoring.
 
 ## Contributors
 
-*This article is maintained by Microsoft. It was originally written by the following contributors.*
+*Microsoft maintains this article. The following contributors wrote this article.*
 
 Principal author:
 
 - [Zoiner Tejada](https://www.linkedin.com/in/zoinertejada) | CEO and Architect
 
+*To see nonpublic LinkedIn profiles, sign in to LinkedIn.*
+
 ## Next steps
 
-- [Pipelines and activities in Azure Data Factory and Azure Synapse Analytics](/azure/data-factory/concepts-pipelines-activities)
-- [Provision the Azure-SSIS integration runtime in Azure Data Factory](/azure/data-factory/tutorial-deploy-ssis-packages-azure)
-- [Oozie on HDInsight](/azure/hdinsight/hdinsight-use-oozie-linux-mac)
+- [Pipelines and activities in Fabric Data Factory](/fabric/data-factory/data-factory-overview)
+- [Provision the Azure-SSIS integration runtime in Data Factory](/azure/data-factory/tutorial-deploy-ssis-packages-azure)
+- [Use Oozie to run a workflow on HDInsight](/azure/hdinsight/hdinsight-use-oozie-linux-mac)
+- [Medallion architecture in Fabric Real-Time Intelligence](https://blog.fabric.microsoft.com/blog/21597)
 
-## Related resources
+## Related resource
 
 - [DataOps for the modern data warehouse](../../databases/architecture/dataops-mdw.yml)