Kleros Etherscan GitHub Workflow

This repository contains scripts for processing and exporting Kleros tags and token data. The scripts interact with the Kleros Curate subgraph to retrieve and process data, which is then exported to CSV files for further analysis or use.

Table of Contents

• Kleros Etherscan GitHub Workflow
  • Table of Contents
  • Installation
  • Usage
    • Individual Data Export
      • Exporting Tags
      • Exporting Tokens
    • Batch Address Tag Query (ATQ)
      • Prerequisites
      • Setup
      • Running the ATQ Process
      • Testing a Specific Module
      • Counting Entries
  • Features
    • Envio Integration (Default)
    • Automatic File Chunking
    • Dual GraphQL Support
  • Project Structure
  • Contributing
  • License

Installation

1. Clone the repository:

   git clone https://github.com/gmkung/kleros-etherscan-batch-process.git
   cd kleros-etherscan-batch-process

2. Install dependencies:

   Ensure you have Node.js installed, then run:

   yarn install

3. Compile TypeScript:

   Compile the TypeScript files to JavaScript:

   tsc

Usage

Individual Data Export

These commands export data directly from the Kleros Curate subgraph using Envio (default) or The Graph.

Exporting Tags

To export Kleros tags, run the following command:

node dist/src/cli-individual-tags-export.js

This will generate CSV files in the dist/output/tags directory, organized by explorer.

Exporting Tokens

To export Kleros tokens, run the following command:

node dist/src/cli-individual-tokens-export.js

This will generate CSV files in the dist/output/tokens directory, organized by explorer.

Note: Both commands use Envio by default. To use The Graph instead:

USE_THEGRAPH=true node dist/src/cli-individual-tags-export.js
USE_THEGRAPH=true node dist/src/cli-individual-tokens-export.js

Batch Address Tag Query (ATQ)

The Address Tags Query (ATQ) process allows for batch retrieval of tags from multiple submodules.

Prerequisites

Before running the ATQ scripts, ensure you have:

• Git
• Node.js (Version 14 or higher recommended)
• Yarn package manager

Setup

1. Create a .env file (optional - Envio is used by default):

   # Optional: Use The Graph instead of Envio
   USE_THEGRAPH=true
   THEGRAPH_API_KEY=your_api_key_here
   
   # Default: Envio (no API key needed)
   # USE_THEGRAPH=false

   Note: The scripts use Envio by default (free, no authentication). Set USE_THEGRAPH=true only if you want to use The Graph instead.

2. Make the scripts executable:

   chmod +x 1_fetch.sh 2_pull_submodules.sh

Running the ATQ Process

1. Fetch and update submodules:

   ./1_fetch.sh && ./2_pull_submodules.sh

2. Export tags to CSV in Etherscan format:

   yarn build && yarn retrieve

   This will generate CSV files in the dist/exports directory.

3. Combined workflow (fetch, update, build, and export):

   ./1_fetch.sh && ./2_pull_submodules.sh && yarn build && yarn retrieve

Testing a Specific Module

To test a specific ATQ module:

./atq_unit_test.sh <git repo URL> <commitID> <chainID>

Example:

./atq_unit_test.sh https://github.com/greentea135/aave-v3-tokens-atq-module.git 2b0edde 1

Counting Entries

To count the total number of contract tags retrieved:

find ./dist/exports -name "*.csv" -type f -exec sh -c 'total=0; for file do count=$(grep -c "" "$file"); echo "$file: $count lines"; total=$((total + count)); done; echo "Total: $total lines"' sh {} +

Features

Envio Integration (Default)

The project uses Envio by default for GraphQL queries:

• ✅ Free & Fast - No API key required
• ✅ Automatic Fallback - The Graph available if needed
• ✅ Clear Logging - Shows which endpoint is being used
• ✅ Single Toggle - Control all queries with one environment variable

Automatic File Chunking

Large exports are automatically split into chunks:

• Max 100,000 lines per file (including header)
• Files named with -part001, -part002, etc. suffixes
• Each chunk includes complete headers
• Prevents issues with extremely large files

Dual GraphQL Support

Switch between providers as needed:

# Use Envio (default - no API key needed)
USE_THEGRAPH=false

# Use The Graph (requires API key)
USE_THEGRAPH=true
THEGRAPH_API_KEY=your_api_key_here

For more details, see ENVIO_MIGRATION.md.

Project Structure

• src/: Contains the source TypeScript files.

  • cli-export.ts: CLI script for exporting Kleros tags.
  • cli-token-export.ts: CLI script for exporting Kleros tokens.
  • kleros-tag-processor.ts: Processes Kleros tags.
  • kleros-token-processor.ts: Processes Kleros tokens.
  • utils.ts: Utility functions used across the project.
  • 3_export-to-csv-etherscan-v2.mts: Script for batch exporting ATQ tags.

• dist/: Contains the compiled JavaScript files.

• output/: Directory where the individual export CSV files are saved.

  • tags/: Contains CSV files with Kleros tags data.
  • tokens/: Contains CSV files with Kleros tokens data.

• dist/exports/: Directory where the batch ATQ CSV files are saved.

Contributing

Contributions are welcome! Please fork the repository and submit a pull request for any improvements or bug fixes.

License

This project is licensed under the MIT License. See the LICENSE file for details.