Finam Hackathon - Stock Price Prediction System

A machine learning system for predicting stock price movements using financial news sentiment and technical indicators. The system combines NLP embeddings from news articles with traditional technical analysis features to forecast multi-horizon returns.

📋 Overview

This project implements a multi-output regression model that predicts stock returns across multiple time horizons (up to 20 days). It leverages:

• News Sentiment Analysis: Extracts semantic embeddings from financial news using sentence transformers
• Technical Indicators: Computes volatility, momentum, moving averages, and volume metrics
• Kalman Filtering: Smooths high-frequency price data for noise reduction
• LightGBM Regressor: Fast gradient boosting for multi-horizon predictions

🚀 Quick Start

Prerequisites

• Python 3.8+
• GPU recommended for sentence transformer embeddings (optional)

📊 Features

News Processing

• Parses aggregated news text with timestamps
• Generates embeddings using multilingual sentence transformers (default: paraphrase-multilingual-MiniLM-L12-v2)
• Fallback to TF-IDF + SVD if GPU unavailable
• Aggregates daily news statistics (mean, std, count)

Technical Features

• Price smoothing: Kalman filtering for noise reduction
• Momentum indicators: 3, 5, 10-day returns
• Volatility metrics: Rolling standard deviation of returns
• Moving averages: Distance from MA (3, 5, 10-day)
• Volume analysis: Log-transformed volume, EMA, volume vs average
• Temporal features: Cyclical encoding of month, day-of-week dummies
• Lag features: Previous 1-3 day prices and volumes

Model Architecture

• Multi-Output LightGBM: Predicts returns for 1-20 days ahead simultaneously
• Regularization: L1/L2 penalties, subsample and feature sampling
• Scalability: Handles multiple tickers with encoder/scaler pipeline

📈 Output

The script generates:

1. regression_model.pkl: Trained multi-output LightGBM model
2. label_encoder.pkl: Ticker encoding mappings
3. scaler.pkl: MinMaxScaler for OHLCV features
4. predictions.parquet: Test set predictions with actual targets
5. per_ticker_predictions_p.csv: Latest forecasts for each ticker (columns: ticker, p1...p20)
6. mae_rmse_by_horizon.png: Visualization of prediction error across horizons

Installation

1. Clone the repository

git clone https://github.com/twirlz-git/finam_hackathon.git
cd finam_hackathon

2. Install dependencies

pip install -r requirements.txt

3. Configure environment variables

Edit the .env file to specify your dataset paths:

# Path to news dataset
NEWS_PATH=/path/to/news_data.csv

# Path to candles (OHLCV) dataset
CANDLES_PATH=/path/to/candles.csv

# Output directory for models and predictions
OUT_DIR=/path/to/output

Or use sed to update paths programmatically:

sed -i 's|NEWS_PATH=.*|NEWS_PATH=/your/path/to/news_data.csv|' .env
sed -i 's|CANDLES_PATH=.*|CANDLES_PATH=/your/path/to/candles.csv|' .env
sed -i 's|OUT_DIR=.*|OUT_DIR=/your/output/directory|' .env

4. Run the pipeline

python3 script.py

🔧 Configuration

Environment variables in .env:

Variable	Default	Description
NEWS_PATH	-	Path to CSV with aggregated news
CANDLES_PATH	-	Path to CSV with OHLCV candlestick data
OUT_DIR	unified_output	Output directory for models/predictions
SBERT_MODEL_NAME	paraphrase-multilingual-MiniLM-L12-v2	Sentence transformer model
TEST_RATIO	0.27	Train/test split ratio
MAX_HORIZON	20	Maximum prediction horizon (days)

📝 Data Format

News Dataset (CSV)

Expected columns:

• ticker: Stock ticker symbol
• all_news_text: Aggregated news text (may contain timestamps like [2024-01-15 10:30:00])

Candles Dataset (CSV)

Expected columns:

• ticker: Stock ticker symbol
• begin: Timestamp for candle start
• open, high, low, close: OHLCV price data
• volume: Trading volume

🧪 Model Performance

The system reports:

• Overall Test MAE: Mean absolute error across all horizons
• Horizon-specific MAE/RMSE: Separate metrics for each prediction day (1-20)

Model was tested on historical data. Achieved MAE 0.47 across all horizons Typical results show prediction accuracy degrades with longer horizons (expected behavior).

🛠️ Technical Stack

• Data Processing: pandas, numpy
• NLP: sentence-transformers, nltk, scikit-learn (TF-IDF, SVD)
• ML: LightGBM, scikit-learn (MultiOutputRegressor)
• Signal Processing: pykalman
• Visualization: matplotlib

📦 Project Structure

finam_hackathon/
├── script.py              # Main pipeline script
├── requirements.txt       # Python dependencies
├── .env                   # Environment configuration
├── weights/              # Directory for model checkpoints (if any)
└── README.md             # This file

This README provides comprehensive documentation for your stock price prediction system, including setup instructions, feature descriptions, and usage examples.[1]

1