Skip to content

paritytech/substrate-archive

master
Switch branches/tags

Name already in use

A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch?
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
August 25, 2022 17:54
June 25, 2021 13:23
August 26, 2019 18:48
May 25, 2020 16:53
August 26, 2022 15:32

Maintainance status: currently unmmaintained; please fork.

Have a look at #464 for more information.

Substrate Archive

Blockchain Indexing Engine

Install the CLIDocumentationContributingFAQ

Rust Matrix

Run alongside a substrate-backed chain to index all Blocks, State, and Extrinsic data into PostgreSQL.

Usage

The schema for the PostgreSQL database is described in the PDF File at the root of this directory.

Examples for how to use substrate-archive are in the examples/ directory

Prerequisites

Extended requirements list found in the wiki

  • depending on the chain you want to index, ~60GB free space
  • PostgreSQL with a database ready for lots of new data
  • Substrate-based Blockchain running with RocksDB as the backend
  • Substrate-based Blockchain running under --pruning=archive

Install The CLI

The CLI

The CLI is an easier way to get started with substrate-archive. It provides a batteries-included binary, so that you don't have to write any rust code. All thats required is setting up a PostgreSQL DB, and modifying a config file. More information in the wiki

The Node-Template CLI

The node-template CLI (in /bin/node-template-archive) is provided as an example of implementing substrate-archive for your chain.

Quick Start

git clone https://github.com/paritytech/substrate-archive.git
# Set up the databases
source ./substrate-archive/scripts/up.sh # Run ./scripts/down.sh to drop the database
cd substrate-archive/bin/polkadot-archive/
# Start the normal polkadot node with `pruning` set to `archive`
polkadot --chain=polkadot --pruning=archive
# Start up the substrate-archive node. `chain` can be one of `polkadot`, `kusama`, or `westend`.
cargo run --release --  -c test_conf.toml --chain=polkadot

You can access the help dialog via cargo run --release -- --help. Note that up and down scripts are meant for convenience and are not meant to be complete. Look in the wiki for more information about the database setup.

FAQ

Contributing

Contributors are welcome!

Read the Doc

Documentation

You can build the documentation for this crate by running cargo doc. More Docs here

Troubleshooting

Archive fails to start with a too many open files error.

Because of the way a RocksDB Secondary Instance works, it requires that all the files of the primary instance remain open in the secondary instance. This could trigger the error on linux, but simply requires that you raise the max open files limit (ulimit):

  • With Docker: $ docker run --ulimit nofile=90000:90000 <image-tag>
  • For Current Shell Session: ulimit -a 90000
  • Permanantly Per-User
    • Edit /etc/security/limits.conf

      # /etc/security/limits.conf
      *           hard    nofile      4096
      *           soft    nofile      1024
      some_usr    hard    nofile      90000
      some_usr    soft    nofile      90000
      insipx      hard    nofile      90000
      insipx      soft    nofile      90000
      root        hard    nofile      90000
      root        soft    nofile      90000
      

For macOS and Linux, a warning message will be raised on the startup when there is a low fd sources limit in the current system, but Windows won't have such a low fd limit warning.

Contact

You can contact us at: