Fuel Data Systems

Official data streaming libraries and tools for the Fuel Network.

📦 Crates   🐛 Report Bug   ✨ Request Feature

📝 About The Project

Fuel Data Systems is a comprehensive suite of libraries and tools designed to enable real-time and historical data streaming and processing from the Fuel Network. This repository houses the official data streaming ecosystem, offering developers a powerful and flexible API to interact with Fuel Network data in real-time.

With Fuel Data Systems, developers can build sophisticated applications that leverage the full potential of the Fuel Network's data, from simple block explorers to complex analytics engines and trading systems.

Getting Started

To get started with local development and syncing blocks locally, see the Syncing Blocks Locally section in the Publisher README.

The CONTRIBUTING.md file contains detailed information about setting up your development environment and contributing to this project.

🚀 Features

• Real-time streaming of Fuel blockchain data
• Historical streaming of Fuel blockchain data
• Support for various Fuel-specific data types
• Customizable filters for targeted data retrieval
• Flexible deliver policies for historical and real-time data
• Seamless integration with other Fuel ecosystem tools

📚 Documentation

To check this stream service up and running, visit:

• https://stream.fuel.network

For the REST API documentation (Mainnet), visit:

• https://api-rest-mainnet.fuel.network/swagger-ui

For codebase documentation, see the README files in the relevant directories:

• Crates Documentation
• Services Documentation
• Cluster Documentation

💻 Fuel Stream Rust SDK

The fuel-streams library provides a simple Rust SDK for connecting to and consuming data from the Fuel blockchain.

Installing

cargo add fuel-streams futures tokio

Basic Usage Example

use fuel_streams::prelude::*;
use futures::StreamExt;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Create a client and establish connection
    let mut client = Client::new(FuelNetwork::Local).with_api_key("your_key");
    let mut connection = client.connect().await?;

    println!("Listening for blocks...");

    // Choose which subjects you want to filter
    let subjects = vec![BlocksSubject::new().into()];

    // Subscribe to blocks with new deliver policy
    let mut stream = connection
        .subscribe(subjects, DeliverPolicy::New)
        .await?;

    while let Some(block) = stream.next().await {
        println!("Received block: {:?}", block);
    }

    Ok(())
}

Filtering Transactions

use fuel_streams::prelude::*;
use futures::StreamExt;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let mut client = Client::new(FuelNetwork::Local).with_api_key("your_key");
    let mut connection = client.connect().await?;

    // Create a subject for script transactions
    let subjects = vec![
        TransactionsSubject::new()
            .with_tx_type(Some(TransactionType::Script))
            .into()
    ];

    // Subscribe to the filtered transaction stream
    let mut stream = connection
        .subscribe(subjects, DeliverPolicy::New)
        .await?;

    while let Some(transaction) = stream.next().await {
        println!("Received transaction: {:?}", transaction);
    }

    Ok(())
}

For more examples and detailed documentation, see the fuel-streams crate documentation.

📑 Architecture Components

Services

Service	Description
API Service	REST API for retrieving blockchain data from an indexed database
Consumer Service	Processes and stores blockchain data in a database
Publisher Service	Subscribes to new blocks and publishes them to a message broker
WebSocket Service	Real-time data streaming via WebSocket connections
Dune Service	Processes blockchain data for analytics with Dune

Libraries

Library	Description
fuel-streams	Main library for streaming Fuel blockchain data
web-utils	Web utilities for building web services in the Fuel ecosystem
domains	Domain models and database infrastructure
subject	Subject derive macro for type-safe subject definitions
types	Core type definitions and utilities
core	Core functionalities and shared components
data-parser	Parser for Fuel blockchain data
message-broker	Message broker implementation for event publishing

Deployment and Infrastructure

Component	Description
Cluster	Deployment infrastructure and configuration
Docker	Docker configuration for local development
Charts	Helm charts for Kubernetes deployment
Scripts	Utility scripts for development and deployment

🛠️ Development

For local development:

1. Setup Environment:

   make create-env
   make setup

2. Start Required Services:

   make start-docker
   make setup-db

3. Run Services:

   • API Service: make run-api
   • Publisher Service: make run-publisher
   • Consumer Service: make run-consumer
   • WebSocket Service: make run-webserver
   • Dune Service: make run-dune

See the CONTRIBUTING.md for more detailed development instructions.

💪 Contributing

We welcome contributions to Fuel Streams! Please check our contributing guidelines for more information on how to get started.

📜 License

This project is licensed under the Apache-2.0 license. See LICENSE for more information.