📈 Financial Research Agent

An AI agent that researches stocks and answers follow-up questions using live financial data, news, and SEC filings.

Live Demo: financial-research-agent.streamlit.app | Built with Claude Code

---

Why This Exists

I built this to answer a question I couldn't find a good answer to: can an LLM agent produce investment briefs that are actually grounded in real sources -- and how would you even know?

The answer required building both the agent and the measurement layer to audit it.

---

What I Measured (and What I Found)

Grounding Eval (LLM-as-judge)

I built an evaluation framework that audits every quantitative and forward-looking claim in each brief against the retrieved source context. A Sonnet judge (temperature 0) labels each claim SUPPORTED, UNSUPPORTED, or INFERENCE.

Early results: 49% unsupported claim rate. Nearly half of what the agent said wasn't backed by anything it retrieved.

After iterating on prompt constraints and forcing generation to stay grounded in source material: 3% unsupported claim rate.

The prompt engineering work -- not the retrieval architecture -- was what actually moved the needle.

Reranking A/B Experiment

I added optional cross-encoder reranking to the RAG pipeline and ran a controlled 4-arm eval across 10 tickers to measure whether it improved grounding:

Arm	Claims	Grounding	Unsupported	Retrieval Latency
Baseline (top-3, no rerank)	66	92.4%	0.0%	4.1s
Plain top-5 (no rerank)	74	86.5%	1.4%	4.5s
Rerank 20→3	84	78.6%	0.0%	20.7s
Rerank 20→5	69	85.5%	0.0%	20.4s

Conclusion: reranking adds 4--5x latency with no reliable grounding benefit. It ships default-off. The measurement framework is the deliverable -- it's what demonstrates the feature isn't needed, rather than assuming it would help.

QLoRA Fine-Tuning Experiment

Can a small local model replace Claude Haiku on section generation at lower cost?

I fine-tuned Qwen2.5-1.5B-Instruct with QLoRA on 104 deterministic, Claude-free training pairs built from real SEC filings and financial data. The fine-tuned model serves 2 of 4 brief sections (Financial Health and Risk Factors); the other two stay on Haiku because deterministic targets couldn't be built for them -- an honest finding about the data, not a gap to paper over.

	Cost per brief	Grounding
Baseline (all Haiku)	$0.00538	88.6%
Hybrid (local model for 2 sections)	$0.00248	85.4%

54% cost reduction at slightly lower grounding. Shipped default-off -- the tradeoff isn't worth it for most users, but the benchmark is there for anyone who needs the cost savings.

I also re-implemented the same fine-tune with a hand-written PyTorch training loop (fine_tune_pytorch_loop.ipynb) -- custom Dataset, manual gradient accumulation and optimizer.step(), hand-written cosine LR, no Hugging Face Trainer. Benchmarked against the Trainer on identical data and config (adamw_torch, cosine schedule, grad-accum 8), the two loss curves track each other closely over 21 optimizer steps -- both start around 1.4--1.5 and trend down together, finishing at 0.50 (native) and 0.35 (Trainer). The curves cross repeatedly, so that final-step gap sits within the run-to-run noise at this scale (~7 optimizer steps/epoch, plus shuffle order and 4-bit-kernel non-determinism) rather than a systematic difference -- confirming the hand-written loop reproduces the Trainer's training dynamics at the gradient-accumulation and optimizer-step level.

Multi-Agent Orchestration Experiment

Does breaking the single agent into a supervisor-orchestrated team improve grounding, or just add cost?

I refactored the brief pipeline into a supervisor graph: a planner decomposes the ticker into SEC retrieval sub-questions and coverage points, a research agent executes the plan against the existing RAG and model-routing code, an inline grounding-critic scores the drafted brief with the same LLM-as-judge used by the offline eval, and a supervisor sends the brief back for revision (bounded at 2 passes) until it clears a grounding threshold. It is flag-gated behind MULTI_AGENT_ENABLED (default off), with the original single-agent pipeline kept as the A/B control. The inline critic and the offline judge share one definition, so there is a single source of truth for grounding.

I ran both paths over the standard 10-ticker set, scored by the same temperature-0 Sonnet judge, with retrieval held at baseline (reranking off, top-3) on both sides so the only differences were the planner-driven queries and the critic loop. Critic threshold: 5% unsupported.

Path	Claims	Unsupported	$/brief	Latency/brief	Revisions/brief
Single-agent (control)	73	1 (1.4%)	$0.0269	26.1s	0.00
Multi-agent	71	0 (0.0%)	$0.0515	43.9s	0.00
Delta		-1.4 pts	+92%	+68%	n/a

Two findings, stated plainly:

1. The critic fired 0 revisions across all 10 real drafts. The base pipeline already drives unsupported claims to roughly 3% overall and effectively 0 on the Executive Summary and Outlook sections this eval scores, so the critic looked at every first draft, found nothing to fix, and passed it. There was no headroom for the revision loop to recover.

2. The one single-agent unsupported claim did not reproduce. Across 73 single-agent claims exactly one was flagged (on WMT). Re-running WMT produced a different draft with zero unsupported claims, confirming the lone flag was temperature-0.2 generation variance, not a systematic weakness in the base synthesis prompt.

The orchestration cost roughly +92% per brief and +68% latency (the planner adds a Haiku call, and the inline critic adds a Sonnet judge to every brief) for a grounding benefit that, on this workload, was null.

Conclusion: it ships default-off. On this workload the multi-agent path showed no grounding benefit because the single-agent baseline was already at the grounding floor, leaving nothing for the critic to recover. That is a statement about this corpus, not a general claim about multi-agent orchestration. The revision loop itself is verified working (an injected bad draft with a fabricated price target is caught by the live judge, sent back, and cleaned on the second pass); it is dormant in practice, not broken. The harness is retained to re-measure if a harder corpus, thinner retrieval, a weaker base model, or longer briefs ever create real grounding headroom. If that happens, a three-way comparison (baseline / planner-only / planner+critic) is the documented next step to attribute any gain to the planner versus the critic. As with the reranking experiment, the deliverable is the measurement that shows when the feature is and is not worth its cost.

---

What It Does

Generate Brief -- enter a ticker and the app produces a structured investment brief:

1. Fetches stock data (yfinance), news (NewsAPI), and SEC filing summaries (EDGAR)
2. Runs two concurrent Pinecone RAG queries to ground the SEC Filing Highlights and Risk Factors sections in actual filing text
3. Generates four middle sections in parallel using Claude Haiku
4. Streams the Executive Summary and Outlook from Claude Sonnet, which receives the pre-written sections as context
5. Caches the completed brief in Redis (exact key research:{TICKER}) and PostgreSQL

Ask a follow-up -- a LangGraph ReAct agent answers free-form questions, selecting whichever tools it needs (stock data, news, SEC filings, or RAG search).

---

Tech Stack

Layer	Technology
LLM -- section generation	Claude Haiku 4.5 (4 sections in parallel)
LLM -- synthesis + ReAct agent	Claude Sonnet 4.6
Agent framework	LangGraph -- create_react_agent (follow-ups) + a supervisor StateGraph (optional multi-agent brief pipeline)
Financial data	yfinance
News	NewsAPI
SEC filings	SEC EDGAR REST API
SEC RAG	LlamaIndex + Pinecone + HuggingFace bge-small-en-v1.5
Reranking (optional)	Cross-encoder BAAI/bge-reranker-base
Observability	LangSmith (tool calls, tokens, latency, cost)
Brief cache	Redis (exact key per ticker, research:{TICKER})
Persistence	PostgreSQL via SQLAlchemy
Async tasks	Celery + Redis
REST API	FastAPI
Frontend	Streamlit
Cloud (backend)	AWS ECS Fargate, RDS PostgreSQL, Secrets Manager, ECR
Infrastructure as Code	Terraform
CI/CD	GitHub Actions → ECR → ECS (OIDC, no static keys)
Development	Claude Code

---

Architecture

Brief pipeline

"Generate Brief"
       │
       ▼
Redis cache (research:TICKER) ──hit──► cached brief
       │ miss
       ▼
get_stock_data  (yfinance)
       │
       ▼
┌──────────────────┐  ┌─────────────────┐
│ get_company_news │  │ get_sec_filings  │  parallel
└──────────────────┘  └─────────────────┘
       │                       │
       └───────────┬───────────┘
                   │
       ┌───────────▼───────────┐
       │   Pinecone RAG (x2)   │  concurrent
       └───────────┬───────────┘
                   │
   ┌───────────────┼───────────────────┐
   ▼               ▼           ▼       ▼
Haiku           Haiku       Haiku   Haiku    4 parallel calls
Financial       Recent      SEC     Risk
Health          Dev.        High.   Factors
   └───────────────┴───────────┴───────┘
                   │
       ┌───────────▼───────────┐
       │  Sonnet: Exec Summary │  streams to browser
       │  + Outlook            │
       └───────────────────────┘

Follow-up questions

"Ask" (free-form question)
       │
       ▼
LangGraph ReAct agent (claude-sonnet-4-6)
  ├─ get_stock_data
  ├─ get_company_news
  ├─ get_sec_filings
  └─ query_sec_filing (Pinecone RAG)
       │
       ▼
     answer

Multi-agent brief pipeline (optional, MULTI_AGENT_ENABLED=true)

The single-agent brief pipeline can be swapped for a supervisor-orchestrated graph. It's off by default -- the single-agent path stays the production default and the A/B control -- and produces the same brief schema and API response, so nothing downstream changes. Toggle the flag to compare the two paths.

"Generate Brief"  (MULTI_AGENT_ENABLED=true)
       │
       ▼
   ┌─────────┐  decomposes the ticker into a research plan: the SEC RAG
   │ Planner │  sub-questions that ground the filing-based sections + coverage
   └────┬────┘
        ▼
   ┌──────────┐ ◄──── revise (critic feedback prepended to the synthesis prompt)
   │ Research │  reuses the EXISTING retrieval + model-routing + synthesis code;
   └────┬─────┘  revision passes re-synthesise Exec Summary + Outlook only
        ▼
   ┌──────────────────┐  the existing LLM-as-judge, promoted to an inline node --
   │ Grounding-critic │  scores the draft for source-grounding (one judge, shared
   └────┬─────────────┘  with the offline eval; `agent/grounding.py`)
        ▼
   ┌────────────┐  unsupported% ≤ CRITIC_MAX_UNSUPPORTED_PCT → done; else send
   │ Supervisor │  back to Research, bounded at MAX_REVISIONS passes
   └────┬───────┘
        ▼
   final brief

• One judge, two callers. The inline critic and the offline grounding eval both call agent/grounding.py:grade_brief() -- there's a single definition of the judge prompt and scoring, not two copies that can drift.
• Schema-safe revisions. Revision passes reuse the already-grounded middle sections and only re-write the Executive Summary + Outlook through the same _synthesis_prompt, so the brief format can't break.
• Bounded loop. MAX_REVISIONS (default 2) caps the critic→research retries; the supervisor accepts the best effort if the budget is exhausted.
• Tracing. Each node (planner / research / critic / supervisor) is its own LangSmith span.

---

AWS Deployment

The FastAPI backend is containerized and runs on AWS ECS Fargate, with a real RDS PostgreSQL database, secrets in AWS Secrets Manager, and a GitHub Actions pipeline that deploys on every push to main. The whole footprint is defined in Terraform (infra/). The Streamlit frontend stays on Streamlit Cloud; Redis/Celery are stubbed in this environment (the cache no-ops and the async endpoint is disabled).

push to main
     │
     ▼
GitHub Actions ──OIDC (no long-lived AWS keys)──► assume scoped IAM role
  1. pytest (CI gate)
  2. docker build → push image (latest + commit SHA) → Amazon ECR
  3. register new task-def revision → update ECS service (wait for stable)
     │
     ▼
ECS Fargate task  (public subnet, public IP, security group locked to my IP)
  FastAPI container (uvicorn, single worker; bge-small model baked into image)
     │                                   │
     ▼                                   ▼
RDS PostgreSQL (t3.micro)        Secrets Manager
  research_briefs table            ANTHROPIC / NEWS / PINECONE / LANGSMITH keys,
  (private, SG-locked to           DATABASE_URL, REDIS_URL — injected as task
   the task's SG)                  env vars by the execution role

Design choices

• Terraform, end to end — ECR, RDS, Secrets Manager, IAM roles, security groups, the ECS cluster/task-def/service, and the GitHub OIDC provider are all in infra/. Local state; terraform.tfvars (with my IP) is gitignored.
• No static cloud credentials — GitHub Actions authenticates via OIDC, assuming a repo-scoped IAM role with just enough permission to push to ECR and deploy the service. Nothing long-lived is stored in the repo.
• Secrets never in the image or git — they live in Secrets Manager and are injected into the task as environment variables at runtime via the execution role.
• Cost-aware — RDS t3.micro on the free tier; Fargate runs in a public subnet with a public IP (no NAT gateway) to avoid NAT cost; the task's security group is locked to a single IP, so the unauthenticated API isn't open to the world.
• Image — python:3.13-slim with the embedding model baked in so cold start doesn't hit the HuggingFace Hub; built in CI (no local Docker needed).

Pausing to save cost

Fargate bills while a task runs, so I park it when I'm not demoing and bring it back for an interview:

infra/ecs-scale.sh 0   # pause  — stop the task (no Fargate compute cost; RDS stays free-tier)
infra/ecs-scale.sh 1   # resume — launch a fresh task (~1-2 min to start)
infra/ecs-ip.sh        # print the running task's public IP + base URL

Or the raw one-liner:

aws ecs update-service --cluster financial-agent-cluster --service financial-agent-api \
  --desired-count 1 --region us-east-1     # 0 to pause

There's no load balancer, so the task gets a new public IP on each resume (infra/ecs-ip.sh fetches it). The service ignores desired_count in Terraform, so scaling this way doesn't fight terraform apply.

See infra/README.md for the apply steps.

---

Running Locally

1. Clone and install

git clone https://github.com/schen9999/financial-agent.git
cd financial-agent
pip install -r requirements.txt

2. Add API keys -- copy .env.example to .env:

Key	Where to get it
ANTHROPIC_API_KEY	console.anthropic.com
NEWS_API_KEY	newsapi.org
REDIS_URL	upstash.com (free tier)
DATABASE_URL	PostgreSQL connection string
PINECONE_API_KEY	pinecone.io (free tier)

3. Run

streamlit run app.py

---

API Endpoints

Method	Endpoint	Description
GET	/health	Health check
POST	/research	Generate brief (synchronous)
POST	/research/async	Submit research job
GET	/research/status/{job_id}	Poll async job status
POST	/ask	ReAct agent answer
GET	/history/{ticker}	Past briefs for a ticker

---

MCP Server

The agent's tools are also exposed over the Model Context Protocol via the official mcp Python SDK (FastMCP), so any MCP client (Claude Desktop, the MCP Inspector, etc.) can call the financial tools over the protocol. This is standalone and additive: mcp_server.py reuses the existing LangChain tools (no duplicated logic) and touches nothing in the FastAPI app, the Streamlit UI, or the agent.

Tools exposed:

MCP tool	Args	Returns
get_stock_data	ticker	price, market cap, P/E, revenue, margins, company info
get_price_history	ticker	12 months of daily closes + percent change
get_company_news	company_name	5 most recent news articles
get_sec_filings	ticker	latest 10-K / 10-Q summaries from EDGAR
query_sec_filings	ticker, question	RAG answer over indexed 10-K / 10-Q text (Pinecone)

Run it

pip install "mcp[cli]==1.28.0"      # already in requirements.txt

mcp dev mcp_server.py               # launches the MCP Inspector over stdio (best for a quick demo)
python mcp_server.py                # raw stdio transport (what Claude Desktop launches)
MCP_TRANSPORT=streamable-http python mcp_server.py   # HTTP transport instead of stdio

Connect Claude Desktop -- add to claude_desktop_config.json (use absolute paths to this repo's venv Python and mcp_server.py), then restart Claude Desktop:

{
  "mcpServers": {
    "financial-research-agent": {
      "command": "/abs/path/financial-agent/.venv/Scripts/python.exe",
      "args": ["/abs/path/financial-agent/mcp_server.py"]
    }
  }
}

Notes

• stdout hygiene (spec compliance). On stdio, stdout is the JSON-RPC channel, so each tool body runs under redirect_stdout(sys.stderr). The transport captures the real stdout once at startup, so library prints (e.g. the RAG pipeline's [rag] ... lines) go to stderr and never corrupt the protocol.
• Windows event-loop fix. On Windows the server forces the asyncio SelectorEventLoop (set at import, before any asyncio/anyio machinery loads). The default ProactorEventLoop makes native-extension HTTP backends -- notably yfinance's curl_cffi/libcurl and the torch/HuggingFace RAG stack -- hang when a tool runs in FastMCP's worker thread over stdio. No effect off Windows.
• Cold start ~13s, dominated by the LangChain import the reused tools pull in. The heavy RAG stack (LlamaIndex + Pinecone + the embedding model) is imported lazily inside query_sec_filings, so it only loads when that tool is first called and the server starts (and runs the other four tools) without a PINECONE_API_KEY.
• Per-call timeout. The reused tools have no request timeout, so a throttled/slow upstream (yfinance, SEC, NewsAPI) would hang the server. Each call is bounded at the wrapper layer by MCP_TOOL_TIMEOUT (default 30s) and fails into the tools' existing {"error": ...} shape. Raise it if the heavy first RAG call (model load + indexing) needs longer.
• Tools need the same keys as the rest of the app (NEWS_API_KEY for news, PINECONE_API_KEY for RAG); they are read from .env.

---

Disclaimer

For informational purposes only. Does not constitute financial advice.