🐊 Rustygator

A modular, high-performance, zero-copy data aggregator written in Rust.

Rustygator is an asynchronous framework designed to route edge data (video, sensors, telemetry) from Sources to Sinks with minimal latency and overhead. It leverages Rust's ownership model to ensure thread safety without the performance penalty of deep-copying data.

🚀 Motivation

1. Zero-Copy Broadcasting: If a generic 4K video frame enters the system, it is allocated once. It is distributed to 10 different consumers (Sinks) via atomic reference counting (Arc). No memory duplication.

2. Async-First: Source to Sink is non blocking

3. Resilience: Rustygator implements "lag detection" and head-dropping strategies to ensure the pipeline keeps moving even if a sink stalls.

4. Reusable and Configurable: Rustygator was originally made because I noticed every project I do tends to be just a core data processor behind a multi source data aggregator.

🏗️ Architecture

The system follows a "Fan-Out" architecture:

Source (Producer) → [MPSC Channel] → Engine Core → [Broadcast Channel] → N x Sinks (Consumers)

• The Envelope: The fundamental unit of data. Wraps a Bytes payload (video/json) with metadata (ID, Timestamp, Headers).

• The Engine: Orchestrates the flow. It spawns independent Green Threads (Tokio Tasks) for every plugin.

📂 Project Structure

This project is organized as a Cargo Workspace:

Crate	Description
crates/rg-core	Defines the Envelope struct, Source and Sink traits, and Mock implementations.
crates/rg-engine	The runtime logic that wires sources to sinks, handles channel broadcasting, and manages backpressure.
crates/rg-common	Shared utilities and error types.

🛠️ Getting Started

Prerequisites

• Rust (Stable)
• Cargo

Running Tests

Since the CLI entry point is currently under development, the best way to verify the system is via the test suite.

1. Run Integration Tests:

Verify that data flows from Dummy Sources to Console Sinks.

Bash

cargo test -- --nocapture

2. Run Performance Benchmarks:

Test the throughput of the Zero-Copy engine on your machine.

cargo test -p rg-engine --release -- --nocapture

🗺️ Roadmap

• Phase 1: Foundation: Workspace setup, Envelope definition, Serde serialization.

• Phase 2: Abstraction: Async Source and Sink trait definitions.

• Phase 3: The Engine: Async runtime implementation using Tokio channels.

• Phase 3.5: Resilience: Implementation of Backpressure handling (Skipping laggy consumers) and Graceful Shutdown.

• Phase 4: "The Brain": YAML-based configuration and CLI entry point.

• Phase 5: Plugins: Real-world implementation (Camera Source, Inference Sink).

• Phase 6: Observability: Structured logging (Tracing) and Metrics.

📄 License

MIT