Skip to content
Unicorns edited this page Jun 23, 2022 · 4 revisions

Subquery golang dictionary generator

Subquery Dictionary Generator

Run alongside a Polkadot or Moonbeam archive node to index all Events, SpecVersions, Extrinsics, EvmTransactions and EvmLogs data into PostgreSQL.

Prerequisites

  • Linux machine
  • Polkadot/Moonbeam or any other substrate running archive node
  • PostgreSQL database
  • config.json file with proper configuration

Install

The project was developed on linux and the following tutorial is for linux:

The rocksdb library was build with the following script:

#!/bin/bash
sudo apt-get install -y libgflags-dev libsnappy-dev zlib1g-dev libbz2-dev liblz4-dev libzstd-dev
pushd /tmp
git clone https://github.com/facebook/rocksdb.git && cd rocksdb
git checkout v6.29.3
export CXXFLAGS='-Wno-error=deprecated-copy -Wno-error=pessimizing-move -Wno-error=class-memaccess'
make static_lib

After that we must set the flags for golang:

go env -w CGO_CFLAGS="-I/tmp/rocksdb/include"
go env -w CGO_LDFLAGS="-L/tmp/rocksdb -lrocksdb -lstdc++ -lm -lz -lbz2 -lsnappy -llz4 -lzstd"

The next step is to start a Polkadot node or a Moonbeam node in archive mode

After we set up the node in archive mode we need to install and configure the Postgres Database. To install it I suggest you use the following commands, but any method you prefer is fine:

sudo apt update
sudo apt install postgresql postgresql-contrib
sudo systemctl start postgresql.service

To create the database use the following commands:

sudo su - postgres
psql
ALTER USER postgres PASSWORD 'myPassword';
CREATE DATABASE database_name;
GRANT ALL PRIVILEGES ON DATABASE database_name TO postgres;
\c database_name;

Now we can create the tables with the following scripts:

  1. If the chain is not evm based (Polkadot), create only the tables from this script.
  2. If the chain is evm based (Moonbeam, Moonriver), create the tables from this script and from this script too.

In the terminal that will start the project you must change the limit for opened files:

ulimit -n 90000

Before running the indexer it is important to edit the configuration file depending on the chain you want to index:

{
    "evm": false, // true for non-evm based chains (Polkadot), false for evm based chains (Moonbeam, Moonriver)
    "version": "1.1.2", // this is the current version of the indexer
    "rocksdb_config": {
        "rocksdb_path": "/tmp/rocksdb-polkadot/chains/polkadot/db/full",  // the location where the node archive is syncronized
        "rocksdb_secondary_path": "/tmp/secondary" // secondary path for the rocksdb instance
    },
    "clients_config": {
        // events are processed slower than the extrinsics so the value "events_workers" should be greater than the "extrinsics_workers"
        "events_workers": 150, // number of events processing workers
        "extrinsics_workers": 100, // number of extrinsics processing workers
        "batch_size": 500 // if this value is too high there will be a lot of RAM used
    },
    "chain_config": {
        "http_rpc_endpoint": "http://127.0.0.1:9933", // the correct http listening address of the node (default is locahost:9933)
        "decoder_types_file": "./network/polkadot.json" // the scale decoder needs the types for the chain defined in this file so it could decode the data correctly
    },
    // change the credentials for the polkadot database in the section "postgres_config"
    "postgres_config": {
        "postgres_user": "postgres",
        "postgres_password": "postgres",
        "postgres_host": "localhost",
        "postgres_port": "5432",
        "postgres_db": "rock",
        "postgres_conn_pool": 20
    }
}

We are using this scale decoder, and it needs predefined structures for data encoded in the chain. To check if they have a file for your chain, look into this folder and set the path to this file in the config.json.

Setup step

To start the indexer you can use the default mode with this config path running the following command:

go run src/main.go

or you can point the path to the custom config.json file using the option --config or short -c:

go run src/main.go --config *path_to_custom_config.json*