🤖 Sentient-Trader

AI-Powered Real-Time Trading Intelligence Platform

Transform financial news into actionable trading signals using AI-powered sentiment analysis and real-time market data

Features • Architecture • Quick Start • API • Contributing

📖 Overview

Sentient-Trader is a production-ready, real-time trading intelligence platform that combines cutting-edge NLP with market data processing to generate actionable trading signals. The system leverages:

🧠 FinBERT - Financial domain-specific BERT model for sentiment analysis
📊 27 Technical Indicators - RSI, MACD, Bollinger Bands, and more
🔄 Distributed Streaming - Kafka (Redpanda) for real-time event processing
💾 Redis Feature Store - Low-latency feature retrieval
🌐 REST API - FastAPI-based endpoints for system interaction

🏗️ System Architecture

Complete system architecture showing data flow from news ingestion to trade signals

🔄 Data Flow Pipeline

┌─────────────────────────────────────────────────────────────────────────────────────────┐
│                              SENTIENT-TRADER PLATFORM                                    │
├─────────────────────────────────────────────────────────────────────────────────────────┤
│                                                                                          │
│   📰 DATA SOURCES                  ⚙️ PROCESSING WORKERS                  🎯 OUTPUT      │
│   ─────────────────               ─────────────────────                 ───────────     │
│                                                                                          │
│   ┌─────────────┐                 ┌─────────────────────┐              ┌───────────┐   │
│   │ Google News │ ──┐             │  Ticker Extraction  │              │   REST    │   │
│   └─────────────┘   │             │  • NLP-based        │              │    API    │   │
│                     │             │  • Confidence Score │              └───────────┘   │
│   ┌─────────────┐   │   Kafka     └─────────┬───────────┘                    ▲         │
│   │  NewsAPI    │ ──┼──────────────────────►│                                │         │
│   └─────────────┘   │  (news-stream)        ▼                                │         │
│                     │             ┌─────────────────────┐              ┌───────────┐   │
│   ┌─────────────┐   │             │ Sentiment Analysis  │              │  Trading  │   │
│   │  yfinance   │ ──┘             │  • FinBERT Model    │              │  Signals  │   │
│   │ (41 stocks) │                 │  • Positive/Neg/Neu │              └───────────┘   │
│   └─────────────┘                 └─────────┬───────────┘                    ▲         │
│         │                                   │                                │         │
│         │   Kafka                           ▼                                │         │
│         │  (market-ticks)        ┌─────────────────────┐              ┌───────────┐   │
│         └───────────────────────►│   Feature Worker    │──────────────│  Strategy │   │
│                                  │  • 27 Indicators    │   Redis      │   Worker  │   │
│                                  │  • RSI, MACD, EMA   │              └───────────┘   │
│                                  └─────────────────────┘                              │
│                                                                                          │
└─────────────────────────────────────────────────────────────────────────────────────────┘

📊 Kafka Topics

Topic	Description	Producer	Consumer
`news-stream`	Raw news articles	News Fetcher	Ticker Worker
`news-with-tickers`	Articles with extracted tickers	Ticker Worker	Sentiment Worker
`news-sentiment`	Articles with sentiment scores	Sentiment Worker	Strategy Worker
`market-ticks`	Real-time stock prices	Market Producer	Feature Worker
`trade-decisions`	Generated trade signals	Strategy Worker	API/Consumers

🚀 Features

Feature	Description
📰 Real-time News Ingestion	Multi-source news aggregation from Google News & NewsAPI
🏷️ Ticker Extraction	NLP-based stock ticker identification with confidence scoring
🎭 Sentiment Analysis	FinBERT-powered financial sentiment classification
📊 Technical Indicators	27 indicators (RSI, MACD, SMA, EMA, Bollinger Bands, etc.)
🤖 Trading Signals	Multi-factor strategy combining sentiment + technicals
📈 Backtesting	Historical strategy evaluation with PnL tracking
🌐 REST API	FastAPI endpoints for real-time data access
🐳 Docker Ready	12-service Docker Compose deployment
☸️ Kubernetes	Production-ready K8s manifests included

📁 Project Structure

Sentient-Trader/
│
├── 📦 docker-compose.yml          # 12-service orchestration
├── 🐳 Dockerfile                  # Container build configuration
├── 📋 requirements.txt            # Python dependencies
├── 🚀 run_full_pipeline.py        # Full system launcher
│
├── 📊 data/                       # Data & configurations
│   ├── company_ticker_map.json    # Company-to-ticker mapping (500+ companies)
│   ├── fetch_real_osint.py        # OSINT data fetcher
│   ├── fetch_real_stock.py        # Stock data fetcher
│   ├── market_ticks.jsonl         # Sample market tick data
│   ├── osint_events.jsonl         # Sample OSINT events
│   └── features/                  # Cached feature files
│
├── 📥 ingest/                     # News Ingestion Adapters
│   ├── google_news_adapter/
│   │   └── fetch_google_news.py   # GNews API integration
│   └── news_api_adapter/
│       └── fetch_news_api.py      # NewsAPI integration
│
├── 🔍 enrich/                     # Data Enrichment
│   ├── sentiment/
│   │   ├── sentiment_analysis.py  # FinBERT sentiment model
│   │   └── worker.py              # Kafka sentiment worker
│   └── ticker_extraction/
│       ├── extractor.py           # NLP ticker extraction
│       ├── worker.py              # Kafka ticker worker
│       ├── confidence.py          # Confidence scoring
│       ├── band_analytics.py      # Band analysis utilities
│       ├── outcome_tracker.py     # Trade outcome tracking
│       └── api.py                 # Extraction API endpoints
│
├── 💾 feature_store/              # Feature Engineering
│   ├── feature_store.py           # Redis feature storage
│   ├── feature_worker.py          # Technical indicator calculator
│   └── api.py                     # Feature API endpoints
│
├── 🧠 tools/                      # Processing Workers
│   ├── strategy_worker.py         # Trade decision engine
│   ├── market_data_producer.py    # yfinance data producer
│   ├── produce_tick.py            # Tick data producer
│   ├── send_test_news.py          # Test news publisher
│   └── send_test_articles.py      # Test article publisher
│
├── 🌐 serve/                      # API Layer
│   └── api.py                     # FastAPI REST endpoints
│
├── 📈 backtester/                 # Backtesting Engine
│   └── run_backtest.py            # Historical strategy evaluation
│
├── 🎯 evaluation/                 # Strategy Evaluation
│   ├── metrics.py                 # Performance metrics
│   └── outcome_labeler.py         # Trade outcome labeling
│
├── 🤖 model/                      # Machine Learning
│   └── train.py                   # Model training scripts
│
├── 🧪 test/                       # Test Suite
│   └── unit/
│       ├── test_sentiment.py      # Sentiment tests
│       └── test_strategy.py       # Strategy tests
│
├── ☸️ infra/k8s/                  # Kubernetes Configs
│   ├── deployment.yaml            # Deployment manifest
│   ├── service.yaml               # Service manifest
│   ├── prometheus-config.yaml     # Prometheus configuration
│   └── grafana.yaml               # Grafana dashboard
│
├── 🛠️ utils/                      # Utilities
│   └── logging_config.py          # Logging configuration
│
├── 📁 results/                    # Output & Results
│   ├── backtest_run.json          # Backtest results
│   ├── ledger_*.csv               # Trade ledgers
│   └── trades_summary_*.csv       # Trade summaries
│
├── 📚 docs/                       # Documentation
│   ├── architecture.png           # System architecture diagram
│   └── architecture.svg           # SVG version
│
└── 📓 notebooks/                  # Jupyter Notebooks
    └── demo.md                    # Demo documentation

🔧 Technical Indicators

The Feature Worker computes 27 technical indicators in real-time:

Category	Indicators
Trend	SMA (5, 10, 20, 50), EMA (12, 26), MACD, Signal Line, MACD Histogram
Momentum	RSI (14), Stochastic %K, Stochastic %D, ROC, Momentum
Volatility	Bollinger Bands (Upper, Middle, Lower), ATR, Standard Deviation
Volume	OBV, Volume SMA, VWAP
Support/Resistance	Pivot Points, Fibonacci Levels

🐳 Quick Start

Prerequisites

Docker & Docker Compose
Python 3.11+ (for local development)
8GB+ RAM recommended

1️⃣ Clone & Setup

# Clone the repository
git clone https://github.com/vikasp07/Sentient-Trader.git
cd Sentient-Trader

# Copy environment template
cp .env.example .env

2️⃣ Configure Environment

Edit .env with your API keys:

# News APIs
GNEWS_API_KEY=your_gnews_api_key
NEWS_API_KEY=your_newsapi_key

# Kafka (Redpanda)
KAFKA_BOOTSTRAP_SERVERS=redpanda:9092

# Redis
REDIS_HOST=redis
REDIS_PORT=6379

# API Settings
API_HOST=0.0.0.0
API_PORT=8000

3️⃣ Launch Services

# Start all 12 services
docker-compose up -d

# View logs
docker-compose logs -f

# Check service health
docker-compose ps

4️⃣ Access the System

Service	URL	Description
REST API	http://localhost:8000	Main API endpoint
API Docs	http://localhost:8000/docs	Swagger documentation
Redpanda Console	http://localhost:8080	Kafka topic browser
MinIO Console	http://localhost:9001	Object storage

🌐 API Reference

Health Check

GET /health

Response:

{
  "status": "healthy",
  "timestamp": "2024-12-25T10:30:00Z"
}

Get Features for Symbol

GET /features/{symbol}

Example:

curl http://localhost:8000/features/AAPL

Response:

{
  "symbol": "AAPL",
  "features": {
    "rsi_14": 65.5,
    "macd": 2.34,
    "sma_20": 178.5,
    "sentiment_score": 0.85
  },
  "timestamp": "2024-12-25T10:30:00Z"
}

Get Trade Decisions

GET /trades

Response:

{
  "trades": [
    {
      "symbol": "AAPL",
      "action": "BUY",
      "confidence": 0.87,
      "sentiment": "positive",
      "price": 178.5,
      "timestamp": "2024-12-25T10:30:00Z"
    }
  ]
}

Submit News Article

POST /news
Content-Type: application/json

{
  "title": "Apple announces record Q4 earnings",
  "content": "Apple Inc reported...",
  "source": "Reuters"
}

🐋 Docker Services

Service	Port	Description
`redpanda`	9092, 8080	Kafka-compatible streaming
`redis`	6379	Feature store
`minio`	9000, 9001	Object storage
`api`	8000	Main REST API
`sentiment-api`	8001	Sentiment service
`sentiment-worker`	-	Sentiment processing
`ticker-extraction-worker`	-	Ticker extraction
`feature-worker`	-	Technical indicators
`feature-api`	8002	Feature API
`strategy-worker`	-	Trade decisions
`news-fetcher`	-	News ingestion
`market-data-producer`	-	Price data

🧪 Testing

# Run all tests
pytest test/ -v

# Run specific test file
pytest test/unit/test_sentiment.py -v

# Run with coverage
pytest test/ --cov=. --cov-report=html

📈 Backtesting

Run historical backtests on your strategy:

# Run backtest with sample data
python backtester/run_backtest.py \
  --data data/market_ticks.jsonl \
  --osint data/osint_events.jsonl \
  --output results/backtest_run.json

☸️ Kubernetes Deployment

Deploy to Kubernetes cluster:

# Apply configurations
kubectl apply -f infra/k8s/

# Check deployment status
kubectl get pods -l app=sentient-trader

# Access logs
kubectl logs -f deployment/sentient-trader

🔒 Trading Modes

The Strategy Worker supports three decision modes:

Mode	Description	Requirements
News-Only	Trade on strong sentiment alone	Sentiment ≥ 0.70
Partial	Trade with basic features	Sentiment + some indicators
Full	Complete multi-factor analysis	All features + sentiment

📊 Technical Stack

Layer	Technology
Language	Python 3.11+
NLP Model	FinBERT (ProsusAI/finbert)
Streaming	Apache Kafka (Redpanda)
Cache	Redis
Storage	MinIO (S3-compatible)
API	FastAPI + Uvicorn
Market Data	yfinance
Containers	Docker + Docker Compose
Orchestration	Kubernetes
Monitoring	Prometheus + Grafana

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🤝 Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

Fork the repository
Create your feature branch (git checkout -b feature/AmazingFeature)
Commit your changes (git commit -m 'Add some AmazingFeature')
Push to the branch (git push origin feature/AmazingFeature)
Open a Pull Request

📧 Contact

Vikas P - @vikasp07

Project Link: https://github.com/vikasp07/Sentient-Trader

⭐ Star this repository if you find it helpful!

Made with ❤️ by Vikas P

Name		Name	Last commit message	Last commit date
Latest commit History 9 Commits
.github		.github
backtester		backtester
data		data
docs		docs
enrich		enrich
evaluation		evaluation
feature_store		feature_store
infra/k8s		infra/k8s
ingest		ingest
model		model
notebooks		notebooks
results		results
serve		serve
test		test
tools		tools
utils		utils
.dockerignore		.dockerignore
.gitignore		.gitignore
Dockerfile		Dockerfile
LICENSE		LICENSE
README.md		README.md
TODO_LEARNING_PHASE.md		TODO_LEARNING_PHASE.md
docker-compose.yml		docker-compose.yml
requirements.txt		requirements.txt
run_full_pipeline.py		run_full_pipeline.py
test_evaluation.py		test_evaluation.py
test_finbert.py		test_finbert.py
test_ticker_extraction.py		test_ticker_extraction.py

Folders and files

Latest commit

History

Repository files navigation

🤖 Sentient-Trader

AI-Powered Real-Time Trading Intelligence Platform

📖 Overview

🏗️ System Architecture

🔄 Data Flow Pipeline

📊 Kafka Topics

🚀 Features

📁 Project Structure

🔧 Technical Indicators

🐳 Quick Start

Prerequisites

1️⃣ Clone & Setup

2️⃣ Configure Environment

3️⃣ Launch Services

4️⃣ Access the System

🌐 API Reference

Health Check

Get Features for Symbol

Get Trade Decisions

Submit News Article

🐋 Docker Services

🧪 Testing

📈 Backtesting

☸️ Kubernetes Deployment

🔒 Trading Modes

📊 Technical Stack

📄 License

🤝 Contributing

📧 Contact

About

Resources

License

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Contributors

Uh oh!

Languages

Packages