Cardano Data Collector

A production-ready Python daemon for collecting Cardano DeFi data from Liqwid Finance and Minswap, storing time-series data in GreptimeDB.

Features

• Multi-Wallet Monitoring - Track multiple Cardano wallets for qToken holdings
• Liqwid Position Tracking - Resolve supply positions with USD valuations
• Minswap Price Collection - High-frequency price data collection
• Time-Series Storage - Persistent storage in GreptimeDB
• Flexible Scheduling - Independent schedules for Liqwid and Minswap data
• Docker Ready - Multi-platform Docker images (amd64/arm64)
• Fail-Fast Configuration - Strict validation with no silent fallbacks

Quick Start

Prerequisites

• Python 3.11+
• GreptimeDB instance
• Cardano wallet addresses to monitor

Installation

# Clone the repository
git clone https://github.com/OthmanB/CardanoDataCollector.git
cd CardanoDataCollector

# Create virtual environment
python -m venv .venv
source .venv/bin/activate  # Linux/macOS
# .venv\Scripts\activate   # Windows

# Install dependencies
pip install -r requirements.txt

Configuration

1. Copy and edit the configuration file:

cp config/config.yaml.example config/config.yaml

2. Set wallet addresses via environment variables:

export LIQWID_MAIN_WALLET_ADDRESS="addr1q..."
export LIQWID_WALLET_OTHER_ADDRESS="addr1q..."

3. Edit config/config.yaml:

wallets:
  - name: "Main Wallet"
    address_env: "LIQWID_MAIN_WALLET_ADDRESS"
  - name: "Other Wallet"
    address_env: "LIQWID_WALLET_OTHER_ADDRESS"

liqwid:
  endpoints:
    liqwid: "https://v2.api.liqwid.finance/graphql"
    koios: "https://api.koios.rest/api/v1"
  schedule:
    interval: "1h"
    retry_interval: "30m"

minswap:
  endpoint: "https://agg-api.minswap.org"
  schedule:
    interval: "20s"
    retry_interval: "10s"

storage:
  greptime:
    host: "http://localhost"
    port: 7010
    database:
      liqwid: "liqwid"
      minswap: "minswap"
    timeout: 10
    create_database: true
  table_asset_prefix: "liqwid_supply_positions_"
  write_batch_size: 500

options:
  fetch_prices: true
  log_level: "INFO"
  max_markets: 200
  fail_fast: false
  retry_attempts: 3
  retry_backoff_seconds: 10

telemetry:
  enable_metrics: false

Running

# Test connections
python main.py --test-connections

# Run once (dry run)
python main.py --once --dry-run

# Run once (with database writes)
python main.py --once

# Run as daemon (continuous)
python main.py

Docker Deployment

Using Pre-built Image

# Pull from GitHub Container Registry
docker pull ghcr.io/othmanb/cardanodatacollector:latest

# Run with environment variables
docker run -d \
  --name cardano-data-collector \
  -v ./config:/app/config:ro \
  -e LIQWID_MAIN_WALLET_ADDRESS="addr1q..." \
  -e LIQWID_WALLET_OTHER_ADDRESS="addr1q..." \
  ghcr.io/othmanb/cardanodatacollector:latest

Using Docker Compose

# Set environment variables in .env file
echo "LIQWID_MAIN_WALLET_ADDRESS=addr1q..." >> .env
echo "LIQWID_WALLET_OTHER_ADDRESS=addr1q..." >> .env

# Start the daemon
docker-compose up -d

Building Locally

docker build -t cardano-data-collector:local .

Architecture

┌─────────────────────────────────────────────────────────────┐
│                   Cardano Data Collector                     │
├─────────────────────────────────────────────────────────────┤
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────────────┐  │
│  │   Liqwid    │  │   Minswap   │  │     Scheduler       │  │
│  │   Client    │  │   Client    │  │  (Independent)      │  │
│  └──────┬──────┘  └──────┬──────┘  └──────────┬──────────┘  │
│         │                │                     │             │
│         ▼                ▼                     ▼             │
│  ┌─────────────────────────────────────────────────────────┐│
│  │              Position Resolution Engine                 ││
│  │  • Match qTokens to Markets                             ││
│  │  • Calculate USD Values                                 ││
│  │  • Aggregate Multi-Wallet Data                          ││
│  └─────────────────────────┬───────────────────────────────┘│
│                            │                                 │
│                            ▼                                 │
│  ┌─────────────────────────────────────────────────────────┐│
│  │                  GreptimeDB Client                      ││
│  │  • Batch Writes                                         ││
│  │  • Table Auto-Creation                                  ││
│  │  • Time-Series Storage                                  ││
│  └─────────────────────────────────────────────────────────┘│
└─────────────────────────────────────────────────────────────┘

Data Collected

Liqwid Positions

Field	Description
timestamp	Collection time (UTC)
wallet_address	Cardano wallet address
asset_symbol	Asset symbol (e.g., ADA, DJED)
qtoken_balance	qToken balance
supply_amount	Underlying supply amount
supply_usd	USD value of supply
exchange_rate	qToken exchange rate

Minswap Prices

Field	Description
timestamp	Price time (UTC)
asset_symbol	Asset symbol
price_ada	Price in ADA
price_usd	Price in USD
volume_24h	24-hour volume

Configuration Reference

See docs/configuration.md for complete configuration options.

Key Configuration Sections

• wallets - Wallet addresses to monitor (via environment variables)
• liqwid - Liqwid API endpoints and schedule
• minswap - Minswap API endpoint and schedule
• storage - GreptimeDB connection and table settings
• options - Runtime options (logging, retries, fail-fast)
• telemetry - Metrics export settings

Documentation

• Installation Guide
• Configuration Reference
• Operation Guide
• API Reference
• Troubleshooting

Development

# Install dev dependencies
pip install -r requirements.txt
pip install pytest pytest-cov

# Run tests
python -m pytest tests/ -v

# Run specific test
python -m pytest tests/test_config.py -v

License

This project is licensed under the MIT License - see the LICENSE file for details.

Contributing

Contributions are welcome! Please read the Development Guide for guidelines.