Flat files decoder for firehose

[][gh-ci]

this crate is designed to decompress and decode headers from binary files, which are called flat files, generated from Firehose. Flat files store all information necessary to reconstruct the transaction and receipt tries. It also checks the validity of receipt roots and transaction roots present in the block headers by recalculating them via the block body data. Details of the implementation can be found here. This check ensures that receipt logs and transaction data stored in the flat files are internally consistent with the block headers also stored in the flat files.

This tool was first presented as a mean to enhance the performance and verifiability of The Graph protocol. However, it turns out it could be used as a solution for EIP-4444 problem of full nodes stopping to provide historical data over one year. The idea is that the flat files that this crate can decode could also be used as an archival format similar to era1 files if the transaction and receipt data stored in the flat files can be verified to be part of the Ethereum canonical history.

For more information, read our ethresear.ch here and our pre-merge solution implementation here to see how we can verify that the flat files are consistent with Ethereum's canonical chain.

Getting Started

Prerequisites

• Rust (stable)
• Cargo (Comes with Rust by default)
• protoc
• Firehose dbin files to decode
  • An example file is provided example0017686312.dbin

Running

Commands

The tool provides the following commands for various operations:

• stream: Stream data continuously.
• decode: Decode files from input to output.
• help: Print this message or the help of the given subcommand(s).

Options

You can use the following options with the commands for additional functionalities:

• -h, --help: Print help information about specific command and options.
• -V, --version: Print the version information of the tool.

NOTICE: either streaming or reading from directory it will verify the receipt root & transaction root matches the computed one for all blocks

Usage Examples

Here are some examples of how to use the commands:

1. To stream data continuously from stdin:

# simply turning on stream stdin reading
cargo run stream

# or from files into stdin
cat example0017686312.dbin | cargo run stream

This will output decoded header records as bytes into stdout

2. To check a folder of dbin files:

cargo run decode --input ./input_files/

This will store the block headers as json format in the output folder. By passing --headers-dir a folder of assumed valid block headers can be provided to compare with the input flat files. Valid headers can be pulled from the sync committee subprotocol for post-merge data.

**NOTICE:**For pre-merge data another approach using the header accumulator is necessary since sync committees will not provide these headers.

Goals

Our goal is to provide The Graph's Indexers the tools to trustlessly share flat files with cryptographic guarantees that the data in the flat files is part of the canonical history of the Ethereum blockchain, enabling Indexers to quickly sync all historical data and begin serving data with minimal effort.

Benchmarking

• Run cargo bench in the root directory of the project
• Benchmark results will be output to the terminal
• Benchmark time includes reading from disk & writing output to disk
• Results can be found in target/criterion/report/index.html

For proper benchmarking of future improvements, fixes and features please compare baselines. Refer to the end of this section of Criterion documentation for more information on creating and comparing baselines.

Testing

Generate code coverage reports with cargo llvm-cov --html and open them with open ./target/llvm-cov/html/index.html.