This is a event driven meta data ingestion tool that I am building with Azure leveraging several of Azure PaaS services.
-
Updated
Dec 11, 2022 - JavaScript
This is a event driven meta data ingestion tool that I am building with Azure leveraging several of Azure PaaS services.
Azure function to set the permission of an Azure Data Lake Store in ARM template deploy (~custom resource)
Azure Data Lake Gen2 storage connectors for Data Culpa - monitor data quality automatically with Data Culpa Validator
This azure function reads multiple files from given datalake folder, deserialize data and merge data from all files together. It can apply filters on data and respond with filtered data in requested format.
Submitting a U-SQL Job to Azure Data Lake Analytics
ETL motor racing data project using Azure Databricks, Pyspark and Azure Date Lakes
Created a movie recommendation system on Azure utilizing Spark SQL by analyzing the MovieLens dataset.
Creates an HDInsight cluster that has an external Hive metastore and access to Azure Data Lake Store
Upload a folder to Azure Data Lake Store
Places a resource lock on your ADLS resources so you cannot accidently delete.
Use Spark with Livy along with Application Insights. Learn to host your external dependencies in data lake.
building a real-world data pipeline in Azure Data Factory (ADF) dataset provided by https://www.ecdc.europa.eu/ ingesting data from sources such as HTTP and Azure Blob Storage into Azure Data Lake Gen2 using ADF. transformed data and loaded transformed data using Databricks Notebook Activity in Azure Data Factory (ADF) and load into Azure Data L…
Databricks ETL Pipeline for retrieving and processing NI TestStand test results, featuring a well-documented notebook for ETL operations, Data Lake for storage, Spark SQL+Python for transformations, and Power BI as the final visualization of factory metrics.
Bulk image streaming and upload using Flink (+ Kubernetes), Kafka, Data Lake, and SQL (Provided with React UI and Node server for Demo).
POC projects working on Cloud Platforms
An application developed to give real-time insights on machine health using Iot sensors by tracking and monitoring parameters such as temperature, pressure, current and humidity.
Add a description, image, and links to the azure-data-lake topic page so that developers can more easily learn about it.
To associate your repository with the azure-data-lake topic, visit your repo's landing page and select "manage topics."