Skip to content

Home

Jump to bottom
Eneko Gonzalez edited this page May 31, 2024 · 8 revisions

ChronoLog Project Overview

ChronoLog is a scalable, high-performance distributed shared log store designed to handle the ever-growing volume, velocity, and variety of modern activity data. It is tailored for applications ranging from edge computing to high-performance computing (HPC) systems, offering a versatile solution for managing log data across diverse domains. Find out more at chronolog.dev

Core Features:

  • Multi-Tiered Storage: Leverages multiple storage tiers (e.g., persistent memory, flash storage) to scale log capacity and optimize performance.
  • High Concurrency: Supports multiple writers and multiple readers (MWMR) for efficient concurrent access to the log.
  • Partial Data Retrieval: Enables efficient range queries for partial log processing, enhancing data exploration capabilities.
  • Total Ordering: Guarantees strict ordering of log entries across distributed environments, eliminating the need for costly synchronization.
  • Synchronization-Free: Employs physical time for log ordering, avoiding expensive synchronization operations.
  • Auto-Tiering: Automatically and transparently moves log data across storage tiers based on age and access patterns.
  • Elastic I/O: Adapts to varying I/O workloads, ensuring efficient resource utilization and performance.

Applicability

ChronoLog is designed to address the challenges of modern applications that generate and process vast amounts of log data. It is particularly well-suited for:

  • Scientific Applications: Astrophysics, geoscience, materials science, cosmology, and more.
  • Internet-of-Things (IoT): Efficiently stores, indexes, queries, and analyzes data from IoT devices.
  • Financial Applications: Supports real-time monitoring, time-series analysis, and fraud protection.
  • HPC Resource Management: Optimizes resource utilization in large-scale HPC systems.
  • System Telemetry and Trace Recording: Captures and analyzes telemetry data from distributed systems.
  • Application Performance Analysis: Facilitates performance monitoring and analysis for complex applications.

Key Innovations

  • Physical Time-Based Ordering: A novel approach to achieving total ordering in distributed systems without synchronization overhead.
  • 3D Log Distribution: A unique model that distributes log data horizontally across nodes and vertically across storage tiers.
  • Decoupled Server-Pull Architecture: Separates event ingestion from event persistence for enhanced performance and scalability.

Publication

A. Kougkas, H. Devarajan, K. Bateman, J. Cernuda, N. Rajesh, X.-H. Sun. "ChronoLog: A Distributed Shared Tiered Log Store with Time-based Data Ordering," Proceedings of the 36th International Conference on Massive Storage Systems and Technology (MSST 2020).

Clone this wiki locally