Exploring the problem of high-scale data ingestion on Azure
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Failed to load latest commit information.


#Data Pipeline Guidance

Microsoft patterns & practices


An updated version of the Cold Storage Processor and the Simulator are available as part of our IoT Journey project. The updated versions are:

Build status

This reference implementation is a work-in-progress project. It is meant to demonstrate proven practices regarding the high-scale, high-volume ingestion of data in a typical event processing system.

The project makes heavy use of Microsoft Azure Event Hubs, a cloud-scale telemetry ingestion service. Familiarity with the general concepts underlying Event Hubs is very useful for understanding the source in this reference implementation.


The two primary concerns of this project are:

  • Facilitating cold storage of data for later analytics. That is, translating the chatty stream of events into chunky blobs.

  • Dispatching incoming events to specific handlers. That is, examining an incoming event and passing it along to an appropriate handler function. The emphasis of our dispatcher solution is on speed and overal throughput.

Next Steps

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact opencode@microsoft.com with any additional questions or comments.