Azure Hands-on Lab - Data Engnieering

1. Create Azure Data Lake Storage Gen 2

Create ADLS gen 2 to store flights data Use Azure Portal to create the service and explorer Create Filesystem

• landingzone
• staging
• clean

2. Create CosmosDB

Create CosmosDB to store some part of data Use query to access data

3. Create Azure Data Facotry

Create ADF to move flight data from blob to ADLS gen 2 Use Copy Data tool to build data pipeline for the first time Use binary copy

4. Create Azure Databricks

Create Azure Databricks and import notebooks (Python) Understand how to transform data

• Read event from EventHub and/or CosmosDB
• Mount ADLS gen 2 and read data0
• Create a table in db (SQL)
• Run data engnieering notebooks

5. Create Azure Machine Learning Service

Create Azure Machine Learning Service Understand how to log runs using Azure ML SDK Install Azure ML SDK on Azure Databricks Simple logging AutoML