vriddhi

वृद्धि — Sanskrit for growth, increase.

A self-hosted NSE end-of-day data pipeline, archive, and screener. Vriddhi downloads every EoD report published by NSE, structures twenty years of market data into a queryable PostgreSQL archive, and serves it through a typed REST API consumed by a React screener dashboard.

Built to replace a fragile, monolithic Dash application with a maintainable, tested, async-first pipeline.

---

What it does

• Downloads all NSE EoD reports daily — CM bhavcopy, F&O bhavcopy, participant OI, market activity, bulk/block deals, short selling, 52-week high/low, index snapshots, ETF data, SPAN files
• Backfills historical data from 2005 using an async concurrent downloader with full NSE holiday awareness
• Parses raw CSVs and ZIP archives into typed, validated DataFrames and bulk-loads them into Postgres using asyncpg's binary COPY protocol
• Exposes a FastAPI REST API with a generated TypeScript client for the frontend
• Provides a React screener dashboard with filters, OHLCV charts, F&O OI, and participant data

---

Architecture

NSE website
    │  httpx.AsyncClient + retry
    ▼
app/ingestion/downloader/     ← async download, no framework coupling
    │  raw files on disk (data/raw/YYYYMMDD/)
    ▼
app/ingestion/parsers/        ← pandas: CSV → typed DataFrame
    │
    ▼
app/ingestion/loaders/        ← asyncpg COPY: DataFrame → Postgres
    │
    ▼
PostgreSQL                    ← Alembic-managed schema, partitioned by year
    │
    ▼
app/api/                      ← FastAPI: SQLAlchemy 2.0 async reads, uvloop
    │  OpenAPI schema → orval → typed axios + TanStack Query hooks
    ▼
frontend/                     ← React + Vite screener dashboard

Dependency flow is strictly one-way: core → db → ingestion → api. Nothing in a lower layer imports from a higher one.

---

Tech stack

Layer	Choice	Why
Runtime	Python 3.14, uv	Fast installs, deterministic lockfile
Event loop	uvloop	libuv-backed, significant I/O throughput gains
HTTP client	httpx	Async-first, HTTP/2, excellent DX
DB driver (writes)	asyncpg + COPY	Fastest bulk insert path into Postgres
ORM (reads)	SQLAlchemy 2.0 async	Type-safe queries for the API read path
Migrations	Alembic	Autogenerate, upgrade + downgrade scripts
Data wrangling	pandas	CSV parsing, type coercion, transform
API framework	FastAPI + Pydantic v2	OpenAPI generation, typed request/response
Frontend	React + Vite
API client	orval + TanStack Query + axios	Generated typed hooks from OpenAPI schema
Charts	Recharts

---

Project structure

vriddhi/
├── backend/
│   ├── app/
│   │   ├── core/           # config, enums, exceptions, logging — no app imports
│   │   ├── db/             # SQLAlchemy models, asyncpg bulk loader, session
│   │   │   └── models/
│   │   ├── ingestion/
│   │   │   ├── downloader/ # session, fetcher, writer, extractor, service
│   │   │   ├── parsers/    # one parser per file key, BaseParser ABC
│   │   │   ├── loaders/    # one loader per table, BaseLoader ABC
│   │   │   └── pipeline.py # IngestionPipeline — ties all three stages together
│   │   ├── api/
│   │   │   ├── routers/    # cm, fo, indices, pipeline
│   │   │   └── schemas/    # Pydantic response models
│   │   └── migrations/     # Alembic versions
│   ├── scripts/
│   │   ├── seed_calendar.py
│   │   └── backfill.py
│   ├── tests/
│   │   ├── unit/
│   │   ├── integration/
│   │   └── api/
│   └── docs/decisions/     # Architecture Decision Records
└── frontend/
    └── src/
        └── client/         # orval-generated, do not edit by hand

---

Getting started

Prerequisites

• Python 3.12+
• uv
• PostgreSQL 15+
• Node.js 20+ (frontend)

Backend

cd backend

# Install dependencies
uv sync

# Configure environment
cp .env.example .env
# edit .env — set DATABASE_URL and RAW_DATA_DIR at minimum

# Run migrations
uv run alembic upgrade head

# Seed the trading calendar (2005 → present)
uv run python scripts/seed_calendar.py

# Start the API
uv run uvicorn app.api.app:create_app --factory --host 0.0.0.0 --port 8000 --loop uvloop

Backfill historical data

# Full backfill from 2005 — takes several hours, resumable
uv run python scripts/backfill.py --from 2005-01-01 --to today --concurrency 8

# Single date
uv run python scripts/backfill.py --date 2024-03-15

Frontend

cd frontend
npm install

# Generate typed API client from running backend
npx orval

# Start dev server
npm run dev

---

Environment variables

Variable	Required	Description
DATABASE_URL	Yes	postgresql+asyncpg://user:pass@host/db
RAW_DATA_DIR	Yes	Absolute path for raw downloaded files
NSE_REQUEST_DELAY	No	Seconds between requests (default 1.5)
NSE_MAX_RETRIES	No	Download retry limit (default 3)
NSE_TIMEOUT	No	HTTP timeout in seconds (default 30)
PIPELINE_CONCURRENCY	No	Backfill concurrent connections (default 8)
LOG_LEVEL	No	DEBUG / INFO / WARNING (default INFO)

---

Data coverage

Market	Source files
Cash Market	CM UDiFF bhavcopy, market activity (MTO), bulk deals, block deals, short selling, 52-week high/low
Futures & Options	F&O UDiFF bhavcopy, participant OI, participant volume, market activity, security ban list, SPAN
Indices	Index snapshots
ETF	ETF EoD via NSE JSON API

Historical coverage starts from 2005-01-01. Pre-July 2024 files use the legacy bhavcopy format; post-July 2024 use NSE's UDiFF format. Both are handled transparently by the parsers.

---

Development

# Run all tests
uv run pytest

# Unit tests only (no DB required)
uv run pytest tests/unit/

# With coverage
uv run pytest --cov=app --cov-report=term-missing

# Type checking
uv run pyright app/

# Regenerate frontend API client (after API changes)
cd frontend && npx orval

Architecture Decision Records

Design decisions are documented in docs/decisions/. Read these before making structural changes — they record not just what was decided but what was explicitly ruled out and why.

---

Caveats

• NSE's website requires browser-like headers and a valid session cookie. The downloader handles cookie acquisition automatically but NSE can change their anti-scraping measures at any time.
• Bulk deals and block deals URLs are undated static files on NSE's server — they always reflect the current day's data. Historical bulk/block deal data must be sourced separately.
• The SPAN file parser exposes metadata only. Raw SPAN binary parsing is out of scope.
• This project is for personal research and educational use. Comply with NSE's terms of service.

---

Acknowledgement

Originally conceived by my father as a personal market research tool after a job loss. This rewrite exists to make that work survive and be maintainable. The URL patterns, file format research, and domain knowledge in nse_downloader.py are entirely his.