This is a event driven meta data ingestion tool that I am building with Azure leveraging several of Azure PaaS services.
-
Updated
Dec 11, 2022 - JavaScript
This is a event driven meta data ingestion tool that I am building with Azure leveraging several of Azure PaaS services.
Azure function to set the permission of an Azure Data Lake Store in ARM template deploy (~custom resource)
This azure function reads multiple files from given datalake folder, deserialize data and merge data from all files together. It can apply filters on data and respond with filtered data in requested format.
Submitting a U-SQL Job to Azure Data Lake Analytics
An Akka Streams source of Azure Data Lake data
Cloud Based Sensoring Solution for flow telemetry. Shown here is an early stage sensoring prototype with azure based alerting and app deployment.
ETL pipeline tailored for Olympics data
Batch process that compacts different parquet files stored at Azure Data Lake Storage following the requirements specified at README.
Places a resource lock on your ADLS resources so you cannot accidently delete.
An application developed to give real-time insights on machine health using Iot sensors by tracking and monitoring parameters such as temperature, pressure, current and humidity.
Databricks ETL Pipeline for retrieving and processing NI TestStand test results, featuring a well-documented notebook for ETL operations, Data Lake for storage, Spark SQL+Python for transformations, and Power BI as the final visualization of factory metrics.
This repository contains code for an end-to-end IoT data pipeline using Azure services. It ingests, processes, and stores IoT device data from AWS S3 to Azure Data Lake Storage and Azure SQL Database, leveraging Azure Data Factory and Azure Functions for seamless integration and automation.
A real-time application to guide cab drivers looking for ride towards the areas of the cities experiencing higher demand
Deploy apache spark in client mode on Kubernetes cluster, integrate with Jupyter notebook through Jupyterhub server.
Data Engineering & Software Blog
Powershell to copy data lake file from local computer
Add a description, image, and links to the azure-data-lake topic page so that developers can more easily learn about it.
To associate your repository with the azure-data-lake topic, visit your repo's landing page and select "manage topics."