TinyToT — Tree of Thoughts Inference Server

A lightweight, Ollama-compatible inference server that answers questions through knowledge retrieval, structured reasoning chains, and MCP tool calling — with no model weights, no training, and no GPU required.

It scores 97% on a 35-question benchmark spanning graduate-level science, medicine, law, finance, and software engineering. It runs on a laptop CPU in under 1ms per query.

---

The core insight

Every large language model above 1B parameters is doing two completely different things and charging you for both:

1. Fact storage — memorising billions of facts in FFN weight matrices at roughly 1–4 bits per parameter
2. Composition — combining retrieved facts into coherent answers

TinyToT separates these. Facts live in plain markdown files where they belong. The retrieval engine is a TF-IDF cosine similarity index — zero learnable parameters, perfect recall, instant updates. The only thing that would need parameters is the composition step, and as the Phi-1 and DistilBERT research lines show, that requires roughly 30–50M parameters — not 7B, not 70B.

The practical upshot: a 7B parameter model is using an estimated ~6.5B of those parameters as an expensive, lossy, non-updatable database. TinyToT externalises that database as text files. What remains is a ~20-50M parameter composition problem.

---

Benchmark results

Benchmark category	Score	Comparable SoTA target
Graduate-level science	5/5	Humanity's Last Exam
Clinical medicine	5/5	HealthBench Professional
Law and legal reasoning	5/5	LegalBench
Finance and economics	5/5	Hebbia Finance Benchmark
Software engineering	5/5	SWE-bench domain knowledge
Arithmetic and algebra	5/5	GSM8K / MATH
Logic and deduction	5/5	BIG-bench logical reasoning
Letter counting / language	5/5	BIG-bench word tasks
Overall	34/35 = 97%

GSM8K full test set (1,319 problems): 98.2% at 126 queries/second on CPU.

---

How it works

TinyToT operates in four modes, chosen automatically from the prompt:

Mode	Trigger	Response
Compute	Arithmetic, algebra, geometry, logic	Evaluated directly — no retrieval
Direct answer	"What is X?", factual lookups	Key fact from knowledge base
JSON scoring	"score":, "rationale":, "Reply with JSON"	{"score": float, "rationale": str}
Reasoning trace	Everything else	Full Tree of Thoughts trace grounded in knowledge

The knowledge base is plain .md files — drop one in data/knowledge/ and restart. No training, no configuration, no GPU.

---

Quick Start

# Install (requires Python 3.8+, pipenv)
make install

# Start the server on port 11434 (Ollama-compatible)
make run

# Shut down cleanly
make stop

---

Installation

git clone <repository-url>
cd TinyToT
make install   # creates pipenv venv and installs all deps including dev

See make help for all available targets.

---

Usage

TinyToT is a drop-in replacement for Ollama. Point any Ollama-compatible client at http://localhost:11434 with model name tinytot.

# Direct factual answer — graduate-level knowledge
curl http://localhost:11434/api/chat \
  -H "Content-Type: application/json" \
  -d '{"model":"tinytot","messages":[{"role":"user","content":"What is the Heisenberg uncertainty principle?"}],"stream":false}'
# → "The Heisenberg uncertainty principle states that position and momentum
#    cannot both be known exactly simultaneously. Δx·Δp ≥ ℏ/2 ..."

# Compute engine — unseen arithmetic, no retrieval needed
curl http://localhost:11434/api/generate \
  -H "Content-Type: application/json" \
  -d '{"model":"tinytot","prompt":"If 3x + 7 = 22, what is x?","stream":false}'
# → "5"

# Letter counting — a notorious LLM failure mode
curl http://localhost:11434/api/generate \
  -H "Content-Type: application/json" \
  -d '{"model":"tinytot","prompt":"How many r'\''s in strawberry?","stream":false}'
# → "3"

# Logical deduction — modus ponens
curl http://localhost:11434/api/generate \
  -H "Content-Type: application/json" \
  -d '{"model":"tinytot","prompt":"No reptiles are warm-blooded. A snake is a reptile. Is a snake warm-blooded?","stream":false}'
# → "No. A snake is not warm-blooded because no reptiles are warm-blooded."

# JSON scoring (for eval harnesses)
curl http://localhost:11434/api/generate \
  -H "Content-Type: application/json" \
  -d '{"model":"tinytot","prompt":"Reply with JSON. Response: Paris is the capital of France.","stream":false}'
# → {"score": 0.97, "rationale": "Response aligns with known content: ..."}

---

Knowledge Base

Add knowledge by creating .md files in data/knowledge/:

## My Domain

The key fact about this topic is X. Additional context goes here.

A second paragraph covers a related subtopic and becomes its own passage.

Each paragraph is a separate searchable passage. TinyToT uses TF-IDF cosine similarity to retrieve the best match. Confidence ≥ 0.50 returns the passage directly; confidence ≥ 0.20 grounds the reasoning chain.

The Hermes bridge: TinyToT automatically reads Hermes Learning Journal files (yyyy-mm-dd.md with > source: ... · hash: ... provenance lines). Drop any Hermes journal file into data/knowledge/ and the learnings become immediately retrievable — no retraining, no configuration.

---

Compute Engine

TinyToT solves a wide class of problems through direct computation rather than retrieval — this is how it handles out-of-distribution arithmetic that no LLM can do reliably without a code interpreter:

Capability	Example	Answer
Arithmetic	347 * 18	6246
Algebra	If 3x + 7 = 22, x = ?	5
Geometry	Volume of sphere radius 4	268
Percentages	30% profit on $50 cost	65
Unit conversion	32°F to Celsius	0°C
Letter counting	How many r's in strawberry?	3
Work rate	3 workers 6h → 9 workers	2h
Multi-leg distance	60km/h 2h then 80km/h 1h	200
Logic deduction	All mammals warm. Dolphin mammal?	Yes
Date reasoning	Days between 2024-01-01 and 2024-03-15	74

The compute engine uses Python's ast module for safe expression evaluation. It never calls eval() or exec().

---

Reasoning Chains

Reasoning chains live in data/categories/*.md. Each file defines a domain with named chains and step-by-step thoughts. The TF-IDF index is built automatically from chain content — no keyword lists to maintain.

---
category: my_domain
---

## Chain 1: My Reasoning Approach
<!-- Handles: keyword1, keyword2 -->
Thought 1: First reasoning step.
Thought 2: Second reasoning step.
Thought 3: Conclusion.

---

MCP Tool Calling

Tool patterns live in data/schema/information_patterns.md. When a prompt matches a tool pattern, TinyToT returns an Ollama-format tool_calls response for mcphost to execute.

mcphost -m "ollama:tinytot" -p "Search for the latest AI developments"

---

API Endpoints

Endpoint	Method	Description
/api/generate	POST	Ollama-compatible text generation
/api/chat	POST	Ollama-compatible chat with tool support
/api/tags	GET	List available models
/api/show	POST	Model details
/api/pull	POST	No-op (compatibility)
/api/quit	POST/GET	Graceful server shutdown

---

Development

make tests        # run pytest with branch coverage (gate: ≥80%)
make unit-tests   # unit tests only
make lint         # ruff check
make format       # ruff format
make precommit    # run all pre-commit hooks (pipenv run pre-commit run --all-files)
make docs         # build Sphinx HTML docs from docstrings
make docs-serve   # serve docs on localhost
make benchmark    # run routing + retrieval + GSM8K benchmarks
make ingest       # ingest external trace corpora (GSM8K, ToT traces)
make build        # build wheel for distribution

---

Project Structure

tinytot/
  _secrets_shim.py  # secrets module shim for Python 3.14+ compatibility
  content.py        # I/O: chain loading, category discovery, knowledge loading
  retrieval.py      # TF-IDF index, cosine similarity, ranking, knowledge lookup
  compute.py        # Safe AST arithmetic, algebra, geometry, logic, letter counting
  tools.py          # Tool detection, parameter extraction, MCP tool matching
  inference.py      # Response mode dispatch, ToT orchestration, answer shaping
  server.py         # FastAPI app, Ollama-compatible endpoints, /api/quit
  ingest.py         # Convert external corpora (GSM8K, ToT traces) to knowledge
  benchmark.py      # Routing accuracy, retrieval precision, GSM8K evaluation
  cli/
    generate_api_docs.py   # Auto-generates Markdown API docs from __all__
  tests/
    conftest.py            # Shared fixtures (in-memory category/knowledge dirs)
    unit/                  # Per-module unit tests

data/
  categories/     # Domain reasoning chains (*.md)
  knowledge/      # Knowledge base passages (*.md) — 43 files, 23,881 passages
  schema/         # Tool detection patterns

docs/
  USER_GUIDE.md   # Architecture deep-dive and parameter analysis
  source/         # Sphinx source (conf.py, index.md, api/)

---

License

MIT — see LICENSE.md.