Big data and visualization

Use case -

Convert unstructured raw data, into more conformed business schema and answer business questions.

We will be ingesting data from two different sources:

• Magento
• Shopify

Creating a data pipeline to store the raw data, process the data and store the data in SnowFlakes as single fact and dimension. Then we will visualize the data to understand the ordering pattern from these two channels.

February 2020

Target audience

• Data engineers
• Data architects

Abstracts

Workshop

At the end of this workshop, you will be better able to build a complete data pipeline and visualize the data. In addition, you will learn how to ingest the data, use Azure Data Factory (ADF) for data movement and operationalizing the data pipeline, how to store the data in raw format, learn about Azure Data Lake Gen 2, Process the data using Databricks ,learn about Databricks auto clustering ,summarize data with Azure Databricks and PySpark, store the data into SnowFlake using Azure DataBricks Connector and visualize the data using Power BI.

Hands-on lab

This hands-on lab is designed to provide exposure to many of Microsoft's transformative line of business applications built using Microsoft big data and advanced analytics.

By the end of the lab, you will be able to show an end-to-end solution, leveraging many of these technologies, but not necessarily doing work in every component possible.

Azure services and related products

• Azure Databricks
• Azure Data Factory (ADF)
• Azure Storage
• Power BI Desktop
• SnowFlake (Optional) can use a SQL Store to store structured data.

Related references