I performed a simple Extract, Transform and Load operation in Azure Databricks. It involved ingesting data from a storage account (Azure Blob Storage), transforming it using pyspark and SQL code in azure databricks and loading the output to another storage account(Azure DataLake Gen2) to be accessed by a visualization tool.
- Mounting Blob Storage & DataLake Gen 2 in Azure Databricks
- Pyspark Cleaning & Transformation in Azure Databricks