Databricks ETL Pipeline for retrieving and processing NI TestStand test results, featuring a well-documented notebook for ETL operations, Data Lake for storage, Spark SQL+Python for transformations, and Power BI as the final visualization of factory metrics.

Platform Extension Framework: Federated Query Engine

A polycloud .NET cloud storage abstraction layer. Provides Blob storage (AWS S3, GCP, FTP, SFTP, Azure Blob/File/Event Hub/Data Lake) and Messaging (AWS SQS, Azure Queue/ServiceBus). Supports .NET 5+ and .NET Standard 2.0+. Pure C#.

An application developed to give real-time insights on machine health using Iot sensors by tracking and monitoring parameters such as temperature, pressure, current and humidity.

A real-time application to guide cab drivers looking for ride towards the areas of the cities experiencing higher demand

This repository contains code for an end-to-end IoT data pipeline using Azure services. It ingests, processes, and stores IoT device data from AWS S3 to Azure Data Lake Storage and Azure SQL Database, leveraging Azure Data Factory and Azure Functions for seamless integration and automation.

Created a movie recommendation system on Azure utilizing Spark SQL by analyzing the MovieLens dataset.

Collection of Databricks and Jupyter Notebooks

Data Engineering & Software Blog

We have dataset of IPL from 2008 to 2020 and we have to visualize analytics on Power BI dashboard. We have to upload that dataset into data lake. After that we have to process that data through pipeline and produce modeled data in warehouse. So, that we will be able to analyze the data in Power BI through pre-defined dashboards.

a high-performance, POSIX-ish Amazon S3 file system written in Go

A list of samples for integration of a .NET application with various Azure cloud services.

An Akka Streams source of Azure Data Lake data

POC projects working on Cloud Platforms

An E2E solution of the Data Resources on Azure using the Snapshot Serengeti dataset. This E2E solution focuses Azure Synapse Analytics, Power Bi & the Azure Data Factory.

A comprehensive guide to understanding and implementing data management and analytics solutions in the Azure ecosystem using Azure Data Fundamentals.

Azure Data Lake Gen2 with azcopy

Fluentd output plugin for Azure Datalake Storage Gen2 (append support)