Research Agent

Research Agent is a small CLI project for turning an open-ended question into a readable research brief from live web results.

The reason this project exists is simple: most "AI research agent" demos look good for one run, but they fall apart as soon as a page blocks scraping, a model response changes shape, or an API returns partial failure. This repo is meant to be a more honest version of that idea. The focus is not just getting text out of a model. The focus is building a research workflow that is inspectable, modular, and worth iterating on in public.

Why Build This

I wanted a project that sits between a throwaway script and a full product:

• Small enough to understand in one sitting.
• Structured enough to grow into a better research tool over time.

The goal for this version is not "perfect autonomous research." It is a solid foundation: search for sources, fetch pages reliably, summarize what was actually retrieved, and make failures visible instead of pretending everything worked.

What It Does

• Takes a research question from the CLI.
• Uses SerpApi to collect live search results.
• Fetches source pages with a browser-style User-Agent.
• Extracts readable text from each page.
• Summarizes each surviving source.
• Builds a final research brief in Markdown or JSON.
• Reports retrieval and synthesis failures explicitly.

Design Principles

• Reliability over demo magic. Failed fetches are recorded, not hidden.
• Modular boundaries. Search, fetch, model, and report logic are separated so they can be replaced independently.
• Testability. The core pipeline can be exercised with mocks and fake providers without spending model credits.
• Honest output. If the system could not retrieve a source, the report should say so.

Project Structure

briefing/
  api/
    app.py
    schemas.py
  cli.py
  config.py
  core/
    agent.py
  domain/
    models.py
  memory/
    store.py
  providers/
    fetch.py
    llm.py
    search.py
frontend/
  src/
tests/
main.py

How It Flows

flowchart LR
    A["Question"] --> B["Search Provider"]
    B --> C["Fetch Provider"]
    C --> D["Per-source Summary"]
    D --> E["Report Synthesis"]
    E --> F["Research Brief"]

Loading

Quickstart

1. Create and activate a virtual environment.
2. Install the dependencies.
3. Add CLAUDE_API_KEY and SERPAPI_KEY to .env.
4. Run a query.

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
python3 main.py "What has OpenAI launched recently?"

You can also run the package directly:

python3 -m briefing "Compare OpenAI and Anthropic"

Or install it in editable mode:

pip install -e .
research-agent "OpenAI enterprise updates"

Useful CLI options:

python3 main.py "Compare OpenAI and Anthropic" --format json
python3 main.py "OpenAI enterprise updates" --output report.md
python3 main.py "Recent AI safety policy changes" --max-sources 3

Testing

The tests are intentionally credit-safe. They use mocks and fake providers only, so they do not call Claude or run the live agent against external services.

python3 -m unittest discover -s tests -v

If you want a deeper explanation of the architecture and the design tradeoffs, see documentation.md.

FastAPI App

The project now also exposes the same pipeline over HTTP with FastAPI.

Run it locally:

cd frontend
npm install
npm run build
cd ..
uvicorn briefing.api.app:app --reload

Useful endpoints:

• GET /
• GET /api/health
• POST /api/research
• GET /api/conversations
• POST /api/conversations/{id}/messages
• interactive docs at GET /docs

The root route now serves a React frontend with:

• a conversation sidebar
• SQLite-backed conversation history
• a message composer
• structured assistant replies from the research pipeline

Example API request:

curl -X POST http://127.0.0.1:8000/api/research \
  -H "Content-Type: application/json" \
  -d '{
    "query": "What has OpenAI launched recently?",
    "max_sources": 3,
    "include_markdown": true
  }'

The API uses the same core pipeline as the CLI. The frontend is now a separate React app in frontend/, and the backend stores conversation history in SQLite so follow-up prompts survive server restarts.

By default the app writes its conversation DB to data/research_agent.sqlite3. You can override that with CONVERSATION_DB_PATH.

Docker

You can run the full app stack with Docker.

Build the image:

docker build -t research-agent:local .

Run it directly:

docker run --rm -p 8000:8000 \
  --env-file .env \
  -e CONVERSATION_DB_PATH=/app/data/research_agent.sqlite3 \
  -v "$(pwd)/data:/app/data" \
  research-agent:local

Or use Compose:

docker compose up --build

That will:

• build the React frontend in the image
• serve the FastAPI app on port 8000
• persist conversation history by mounting ./data into the container

CI/CD

The repo now includes two GitHub Actions workflows:

• ci.yml runs backend tests, builds the React frontend, and verifies the Docker image builds on pushes to main and on pull requests.
• docker-publish.yml builds and pushes a container image to ghcr.io/<owner>/<repo> on pushes to main, version tags like v1.0.0, and manual dispatches.

If you enable GitHub Container Registry for the repo, the publish workflow is enough to give you a basic CD path for container deployments.

What Changed In Phase 1

This version fixes the main problems from the original prototype:

• Anthropic text blocks are parsed correctly instead of printing TextBlock(...).
• Requests use a browser-style User-Agent, which reduces basic 403 failures.
• Failed fetches are tracked as failures instead of being summarized as page content.
• The code is organized into core, domain, and providers modules.
• The CLI can emit either Markdown or JSON output.
• The pipeline is testable without paying for model calls.

Current Limitations

• Search currently depends on SerpApi only.
• Content extraction is still lightweight and could be improved with a stronger readability layer.
• Final synthesis quality still depends on the model and the quality of the retrieved sources.
• There is no caching yet, so repeated runs will hit the same services again.
• Conversation memory is persistent, but it is still single-node local storage rather than a multi-user production setup.

Roadmap

• Add caching for fetched pages and model outputs.
• Improve extraction for noisy articles and docs pages.
• Add richer citations in the final brief.
• Save reports to an examples/ or reports/ folder for showcase demos.
• Add CI so the repo is easier to trust at a glance.