Ethereum Transaction Indexer

A robust Ethereum transaction indexer that processes and stores blockchain data, providing a rich set of APIs for querying historical metrics and transaction data.

Features

Real-time blockchain data indexing
Historical metrics and analytics
RESTful API endpoints for data access
Support for both forward and backward indexing
Efficient batch processing
PostgreSQL database backend
Configurable time ranges for metrics

Prerequisites

Node.js (v14 or higher)
PostgreSQL database
Access to an Ethereum node (JSON-RPC endpoint)

Configuration

The indexer can be configured using environment variables:

JSON_RPC_URL=http://localhost:8545
START_BLOCK=0
POLL_INTERVAL_MS=5000
BLOCK_DATA_DIR=./data/blocks
STATE_FILE_PATH=./data/indexer_state.json
RPC_MAX_RETRIES=5
RPC_RETRY_DELAY_MS=1000
LOG_LEVEL=info
STATUS_SERVER_PORT=3001
DATABASE_URL=postgresql://user:password@host:port/database
CLEAR_DATA_ON_START=false
BACKWARD_SYNC_BATCH_SIZE=10

API Endpoints

All endpoints support the following time ranges:

1h: Last hour
6h: Last 6 hours
1d: Last 24 hours
7d: Last 7 days
30d: Last 30 days

Indexer Status

Get the current status of the indexer, including forward and backward sync progress.

GET /status

Response:

{
  "status": "running",
  "latestBlockOnChain": 12345678,
  "currentForwardHead": 12345670,
  "currentBackwardHead": 12345600,
  "backfill": {
    "complete": false,
    "active": true,
    "blocksProcessed": 1000,
    "blocksRemaining": 500,
    "percentageComplete": 66.67,
    "startTime": 1678900000,
    "elapsedTimeMs": 3600000,
    "elapsedTimeFormatted": "1h 0m 0s",
    "blocksPerSecond": 0.28,
    "estimatedTimeRemainingMs": 1800000,
    "estimatedTimeRemainingFormatted": "30m 0s",
    "etaTimestamp": "2024-03-15T12:00:00.000Z"
  },
  "config": {
    "startBlock": 0,
    "pollIntervalMs": 5000,
    "clearDataOnStart": false
  }
}

Transaction Count

Get the number of transactions in a time range.

GET /transactions/count?range=1d

Response:

{
  "range": "1d",
  "interval": "1 day",
  "count": 150000
}

Transaction Value

Get the total value transferred in a time range.

GET /value?range=7d

Response:

{
  "range": "7d",
  "interval": "7 days",
  "totalValue": "1000000000000000000"
}

Gas Usage

Get gas usage statistics for a time range.

GET /gas?range=1d

Response:

{
  "range": "1d",
  "interval": "1 day",
  "totalGasUsed": 1500000000,
  "averageGasPerBlock": 15000000,
  "totalBlocks": 100
}

Transaction Types

Get the distribution of transaction types in a time range. For contract interactions, the input data is decoded as UTF-8 and parsed as JSON. If the JSON contains an internalTXType field:

6 → stake
7 → unstake
any other value → other_contract_interaction If the input data is not JSON or does not contain internalTXType, it is classified as contract_interaction.

GET /transactions/types?range=7d

Example response:

{
  "range": "7d",
  "interval": "7 days",
  "distribution": {
    "transfer": 1000,
    "stake": 120,
    "unstake": 30,
    "other_contract_interaction": 15,
    "contract_interaction": 50
  }
}

Top Accounts

Get the top accounts by net value in a time range.

GET /accounts/top?range=30d&limit=20

Response:

{
  "range": "30d",
  "interval": "30 days",
  "topAccounts": [
    {
      "address": "0x123...",
      "netValue": "1000000000000000000"
    },
    // ... more accounts
  ]
}

Block Metrics

Get block production statistics for a time range.

GET /blocks/metrics?range=1h

Response:

{
  "range": "1h",
  "interval": "1 hour",
  "totalBlocks": 7200,
  "uniqueMiners": 50,
  "averageTransactionsPerBlock": 150,
  "averageBlockTime": 12.5,
  "timeRange": {
    "start": 1678900000,
    "end": 1678986400
  }
}

Transaction Details

Get details for a specific transaction.

GET /transactions/0x123...

Response:

{
  "tx_hash": "0x123...",
  "block_number": 12345678,
  "tx_index": 0,
  "from_address": "0xabc...",
  "to_address": "0xdef...",
  "value": "1000000000000000000",
  "gas": 21000,
  "gas_price": "20000000000",
  "input_data": "0x",
  "nonce": 0
}

Address Transactions

Get transactions for a specific address.

GET /addresses/0x123.../transactions

Response:

{
  "address": "0x123...",
  "transactions": [
    {
      "tx_hash": "0xabc...",
      "block_number": 12345678,
      // ... transaction details
    },
    // ... more transactions
  ]
}

Block Details

Get details for a specific block.

GET /blocks/12345678

Response:

{
  "block_number": 12345678,
  "block_hash": "0x123...",
  "parent_hash": "0xabc...",
  "timestamp": 1678900000,
  "miner": "0xdef...",
  "gas_used": 15000000,
  "gas_limit": 30000000,
  "size": 1000,
  "transaction_count": 150
}

Time-Range Statistics

Get comprehensive statistics for a time range.

GET /stats?range=24h

Response:

{
  "range": "24h",
  "startTime": 1678900000,
  "endTime": 1678986400,
  "totalTransactions": 150000,
  "totalValue": "1000000000000000000",
  "totalGasUsed": 1500000000,
  "uniqueAddresses": 50000
}

Unique Account Count

Get the total number of unique accounts in the indexer.

GET /stats/accounts/unique-count

Response:

{
  "uniqueAccountCount": 12345
}

Error Responses

All endpoints return standard HTTP status codes and error messages in the following format:

{
  "error": "Error message description"
}

Common status codes:

200: Success
400: Invalid request parameters
404: Resource not found
500: Internal server error

Development

Clone the repository
Install dependencies:
```
npm install
```
Set up environment variables
Start the indexer:
```
npm start
```

Deployment Guide

Prerequisites

Google Cloud Platform (GCP) account
Google Cloud SDK installed
Docker installed
Node.js and npm installed

Local Development Setup

Clone the repository:

git clone <repository-url>
cd transaction-indexer

Install dependencies:
```
npm install
```

Set up environment variables:

cp .env.example .env
# Edit .env with your configuration

Start the application locally:
```
npm start
```

Docker Local Testing

Build and run using Docker Compose:
```
docker-compose up --build
```
Access the API at http://localhost:3000

GCP Deployment

1. Initial Setup

Create a new GCP project:

gcloud projects create [PROJECT_ID]
gcloud config set project [PROJECT_ID]

Enable required APIs:

gcloud services enable \
  cloudbuild.googleapis.com \
  run.googleapis.com \
  containerregistry.googleapis.com

Create a Cloud SQL instance for PostgreSQL:

gcloud sql instances create transaction-indexer-db \
  --database-version=POSTGRES_15 \
  --tier=db-f1-micro \
  --region=us-central1

Create a database:

gcloud sql databases create indexer --instance=transaction-indexer-db

Create a database user:

gcloud sql users create indexer \
  --instance=transaction-indexer-db \
  --password=[PASSWORD]

2. Configure Cloud Build

Set up Cloud Build trigger:

gcloud builds triggers create github \
  --repo-name=[REPO_NAME] \
  --branch-pattern="^main$" \
  --build-config=cloudbuild.yaml

Set up Cloud Build variables:

gcloud builds triggers update [TRIGGER_ID] \
  --substitutions=_DATABASE_URL="postgresql://indexer:[PASSWORD]@/indexer?host=/cloudsql/[PROJECT_ID]:us-central1:transaction-indexer-db"

3. Deploy to Cloud Run

Build and push the container:

gcloud builds submit --config cloudbuild.yaml

Deploy to Cloud Run:

gcloud run deploy transaction-indexer \
  --image gcr.io/[PROJECT_ID]/transaction-indexer \
  --platform managed \
  --region us-central1 \
  --allow-unauthenticated \
  --set-env-vars="DATABASE_URL=[DATABASE_URL],LOG_LEVEL=info,START_BLOCK=0,POLL_INTERVAL_MS=1000,CLEAR_DATA_ON_START=false,STATUS_SERVER_PORT=3000"

4. Monitoring and Maintenance

View logs:

gcloud logging read "resource.type=cloud_run_revision AND resource.labels.service_name=transaction-indexer"

Monitor metrics in Cloud Console:
- Navigate to Cloud Run > transaction-indexer
- View metrics, logs, and revisions

Scale the service:

gcloud run services update transaction-indexer \
  --min-instances=1 \
  --max-instances=10

Production Considerations

Database Backup
- Set up automated backups for Cloud SQL
- Configure backup retention period
- Test restore procedures
Security
- Use Secret Manager for sensitive data
- Enable VPC Service Controls
- Configure IAM roles and permissions
- Enable Cloud Audit Logs
Monitoring
- Set up Cloud Monitoring alerts
- Configure uptime checks
- Monitor resource usage
Scaling
- Configure auto-scaling based on load
- Set appropriate resource limits
- Monitor performance metrics
Cost Optimization
- Use appropriate machine types
- Monitor resource usage
- Set up budget alerts

Troubleshooting

Database Connection Issues
- Verify database credentials
- Check network connectivity
- Review Cloud SQL logs
Deployment Failures
- Check Cloud Build logs
- Verify environment variables
- Review container logs
Performance Issues
- Monitor resource usage
- Check database performance
- Review application logs

For more detailed information about any of these steps, refer to the GCP documentation.

License

MIT

Running Multiple Environments (Testnet & Mainnet)

This project supports running separate indexers for testnet and mainnet, each with its own configuration and database. This is achieved using separate Docker Compose files and environment files.

1. Environment Files

.env.testnet — configuration for testnet (uses https://api-testnet.shardeum.org)
.env.mainnet — configuration for mainnet (uses https://api.shardeum.org)

Each file sets its own DATABASE_URL and other environment variables.

2. Docker Compose Files

docker-compose.testnet.yml — runs the testnet indexer and a dedicated Postgres instance
docker-compose.mainnet.yml — runs the mainnet indexer and a dedicated Postgres instance

3. How to Run

Testnet:

docker-compose -f docker-compose.testnet.yml up --build

API available at http://localhost:3001
Database on port 5433

Mainnet:

docker-compose -f docker-compose.mainnet.yml up --build

API available at http://localhost:3002
Database on port 5434

You can run both at the same time since they use different ports and database volumes.

4. Customization

Edit .env.testnet and .env.mainnet to adjust settings for each environment.
You can further customize the compose files for scaling, resource limits, etc.

Name		Name	Last commit message	Last commit date
src		src
.env.mainnet		.env.mainnet
.env.testnet		.env.testnet
.gitignore		.gitignore
Dockerfile		Dockerfile
README.md		README.md
cloudbuild.yaml		cloudbuild.yaml
docker-compose.mainnet.yml		docker-compose.mainnet.yml
docker-compose.testnet.yml		docker-compose.testnet.yml
docker-compose.yml		docker-compose.yml
package-lock.json		package-lock.json
package.json		package.json
tsconfig.json		tsconfig.json

Latest commit

History

Folders and files

Repository files navigation

Ethereum Transaction Indexer

Features

Prerequisites

Configuration

API Endpoints

Indexer Status

Transaction Count

Transaction Value

Gas Usage

Transaction Types

Top Accounts

Block Metrics

Transaction Details

Address Transactions

Block Details

Time-Range Statistics

Unique Account Count

Error Responses

Development

Deployment Guide

Prerequisites

Local Development Setup

Docker Local Testing

GCP Deployment

1. Initial Setup

2. Configure Cloud Build

3. Deploy to Cloud Run

4. Monitoring and Maintenance

Production Considerations

Troubleshooting

License

Running Multiple Environments (Testnet & Mainnet)

1. Environment Files

2. Docker Compose Files

3. How to Run

4. Customization

About

Resources

Stars

Watchers

Forks

Releases

Packages

Used by

Contributors

Languages