Solana Token Data Ingestor

Description

This project, which is in progress, is an G/RPC listener that listens for token transactions on the Raydium Constant Product AMM on the Solana Blockchain. The ingestor listens for swap transactions and mints along with burns if enabled. It continuously listens for these events and computes price, volume, and holdings changes; constantly caching then streaming the data via Kafka. From there, the data is then served through a RESTful API to trading and analytical microservices. The token data processing is done in golang for speed, the client API is built using FastAPI, and the G/RPC listener is in pure python. The database used is TimescaleDB and Interservice communication is done via Redis Pub/Sub and Kafka. All microservices along with Redis, Redis Commander, Kafka-UI, and TimescaleDB are currently being deployed using Docker Compose. It is recommended to host TimescaleDB, Kafka, and Redis on a cloud provider such as AWS, GCP, or Azure in production. For lowest latency, it is recommended to host the services in the same region/data center.

Tech Stack

Icon	Name	Description
	Python	For the G/RPC listener and API
	FastAPI	Framework for the API
	Go	For token data processing
	Redis	For caching and pub/sub
	Kafka	Persistence Queue
	Docker	For containerization
	TimescaleDB	Time series database for token data
	PostgreSQL	For persistent storage (Migrated to TimescaleDB)
	Prisma	As the ORM (Removed during migration)

Updated Program Flow Diagram

Bundled Services & Default Ports

• Redis :6379 Default
• Redis Commander :8080 Default
• Kafka :9092 Default
• Kafka-UI :8081 Default
• TimescaleDB :5432 Default

SIMPLE INSTALLATION

Clone the repository

git clone https://github.com/Zaydo123/Solana-RPC-Listener

Go to the project directory

cd Solana-RPC-Listener

Create Environment Variables

In each microservice directory, there should be a .env.example file. Copy the contents of the .env.example file and create a new .env file in the same directory. Fill in the values with your own.

Build and Run the Docker Containers

docker-compose up --build

MANUAL INSTALLATION

Dependencies for the Python API and RPC Listener

In the microservice directory, run the following command to install the dependencies:

pip install -r requirements.txt

Dependencies for the Go Token Data Processor

In the token-data-processor directory, run the following command to install the dependencies:

go mod tidy
go mod download

then you may build the binary using

go build -tags musl -o myapp cmd/token-processor/main.go

Production run command for Logger/API service

production run command

gunicorn -k uvicorn.workers.UvicornWorker main:app

development run command - configured with uvicorn

python main.py

Common Issues

1. Pyscopg2 not installing on MacOS
   1. Install Homebrew
   2. Install openssl - brew install openssl
   3. If Processor >= M1 Chip:
      • export LIBRARY_PATH=$LIBRARY_PATH:/opt/homebrew/opt/openssl/lib - Adds openssl to the library path
   • Intel Macs:
     • export LIBRARY_PATH=$LIBRARY_PATH:/usr/local/opt/openssl/lib/ - Adds openssl to the library path
   4. pip install psycopg2
2. Docker build failing on MacOS relating to confluent-kafka
   1. Instead of running barebones go build, run the following command:
      • go build -tags musl -o myapp cmd/token-processor/main.go
      • See more at confluentinc/confluent-kafka-go#805

Notes

• Events API / Logger is still in progress.
• Database automatically creates tables and columns on first run.
• Migration to TimescaleDB is in progress.
• The project is still in progress and is not yet complete.
• Trading services are not being shared in this repository.