Geoharness

A minimal ReAct agent built in pure Python for geospatial tasks: currently evaluated using solar siting, but not explcitly tuned for solar siting. Built using Python, avoiding frameworks where possible.

Live demo: Here — eval results with full agent reasoning traces

A few overarching goals:

• Understand how basic/unengineered an agent architecture can be whilst still being performant (no strict graph traversals or large tool sets).
• How well a different modality of data (geospatial) can be handled by LLMs — can an agent answer geospatial questions and understand the context.
• Build a local MCP server for the agent's tools to evaluate the drawbacks and benefits of MCP in practice (e.g. if the MCP communication layer affects performance).

Current target domain question: Is this a good coordinate to place solar panels?

Motivation

Commercial AI products I have built have used a graph-based architecture in the quest for robustness and repeatability, using a large set of tools and planners for predictable tool calling.

As graph-based products grow, I feel that the maintenance overhead of the graph, the increasing difficulty of extensibility and the brittleness of tasks which fail prescribed traversal started to become less worth it — especially as these architectures seemed to constrain the power afforded by newly released models.

The alternative — fewer, more primitive tools with stronger models — is worth testing properly. I wanted to use a basic ReAct structure with very few tools to evaluate how well a simple agent architecture can accomplish complex tasks. Geospatial data seemed relatively complex as a test bed. The pure Python stipulation is a bit of extra fun, and useful for understanding how popular frameworks abstract away agent architecture.

Architecture

A single ReAct loop (act → observe → reflect) with a small set of primitive tools. The agent calls tools in parallel where independent, reflects on its own progress, and commits to a structured verdict at the final output step.

Tools

Tool	Source	What it returns
get_climate_data	NASA POWER	Monthly solar irradiance and temperature averages for a coordinate
get_terrain_data	OpenTopography (SRTM GL1 / COP30)	Elevation, slope, and aspect for a ~1km radius around a coordinate
web_search	DuckDuckGo	Broad web search (excluded from eval runs)
web_fetch	HTTP scrape	Full text of a specific URL

Key implementation decisions

• Parallel tool calls — climate and terrain data are fetched concurrently via ThreadPoolExecutor, halving tool-call latency
• Structured output on the final step only — output_config with JSON schema is scoped to agent.output() only; act, observe, and reflect use plain text formats that the parser depends on
• Disk cache for all tool calls — terrain data is static, climate data is highly cacheable; round-trip to external APIs only happens on cache miss
• Tool error detection — errors from OpenTopography or NASA POWER are flagged in eval results without crashing the run

MCP server

The geospatial tools are exposed as an MCP server via FastMCP, making them available to any MCP-compatible client including Claude Code.

# stdio (for Claude Code integration)
uv run geo_mcp.py

# HTTP (for eval or external clients)
uv run geo_mcp.py --http

To use with Claude Code, add the following to .claude/settings.json in the project root:

{
  "mcpServers": {
    "geoharness": {
      "command": "uv",
      "args": ["run", "geo_mcp.py"],
      "cwd": "/path/to/geoharness"
    }
  }
}

Once configured, the server will start automatically when you open the project. You can then ask Claude Code directly: "Is Seville a good location for solar panels?" and it will call the tools natively.

MCP eval finding: running the (then 11-location) eval via the MCP path (tools called through the protocol rather than directly) produced the same score with the same failures as the direct tool path. This confirms the quality is in the tool data, not the custom ReAct prompt tuning.

Eval

A hand-labelled eval set of 45 locations tests whether the agent reaches the correct GOOD / MARGINAL / BAD verdict from raw geospatial data alone. Current score: 35/45 (78%).

Scoring rules are documented in eval/solar/eval_rules.md. The set spans latitudes, hemispheres, and failure modes (low irradiance, high cloud cover, bad aspect, steep slope, extreme temperature). Ground-truth labels are derived mechanically from the scoring rules against raw tool data — no model or human judgement is involved in labelling, so the eval can't be biased by training data. web_search is excluded — the agent must reason from tool data alone, not training knowledge.

The dominant remaining failure mode is calibration rather than process: the agent's sense of what counts as a "good" irradiance, slope, or temperature value doesn't always match the rubric's thresholds.

# standard eval
uv run eval/solar/eval.py

# eval via MCP server
uv run eval/solar/eval.py --mcp

Results are saved to eval/solar/results/ with a UTC timestamp per run and include the full Langfuse trace ID for each case.

Architectural baseline

To test whether the ReAct loop (act → observe → reflect) is actually earning its keep, eval/solar/eval_baseline.py runs the same 45 locations through a single LLM call: climate and terrain data are pre-fetched and embedded directly in one prompt, with the same task framing and output schema as the ReAct eval — no tools, no loop.

Baseline score: 32/45 (71%) vs. ReAct's 35/45 (78%).

ReAct comes out ahead, but not by a clean sweep — 9 of the 45 locations flip outcome between the two modes (6 in ReAct's favour, 3 against), and 7 locations fail in both modes. The shared failures point to a calibration ceiling that the extra reasoning steps don't address; the loop mostly reshuffles which cases it gets right rather than uniformly improving on the baseline.

uv run eval/solar/eval_baseline.py

Running locally

1. Copy .env.example to .env and fill in your credentials
2. uv sync to install dependencies
3. uv run main.py "[your query here]"

Required environment variables:

ANTHROPIC_KEY=
LLM_MODEL=
LANGFUSE_PUBLIC_KEY=
LANGFUSE_SECRET_KEY=
LANGFUSE_BASE_URL=https://cloud.langfuse.com
OPENTOPOGRAPHY_KEY=

Observability

Each agent run is traced end-to-end using Langfuse. The full loop appears as a single trace named Geoharness react, with each LLM call (act, observe, reflect, output) as a labelled child generation — useful for inspecting prompt inputs, model outputs, and token usage at each step. These traces are also used for collecting outputs for each LLM call, useful for displaying on the demo.

Frontend

A static frontend displays the eval results with the full agent reasoning trace per location. Built with vanilla HTML/JS.

# develop locally
cd frontend && python3 -m http.server 3000