Open-source platform to measure, test, and see inside AI agents.
Writing an agent is easy now. The hard part is proving it works well. AgentLens fills exactly that gap.
Wiki · Roadmap · Architecture · Contributing
This repository is the scaffolding for a 6-month, build-it-yourself program. The application — the two services, the database schema, the agent loop, the eval harness — is not included on purpose: building it, month by month, is the whole point. This repo holds the planning, documentation, and GitHub workflow that the two of us build on top of. Start at Wiki › Getting Started and Month 1 — Plan.
Writing an AI agent takes a week today. But these questions have no answer:
- Did the agent actually complete this task correctly?
- Did yesterday's prompt change improve or break the system?
- Why did the agent make this decision? Which tool did it pick, and why?
- How much did this task cost, how many seconds, how many steps?
There are plenty of open-source tools that build agents. There is almost none that measures an agent and shows what's inside it.
AgentLens does two things at once:
- A working agent platform — a multi-step, tool-calling agent runtime built to production hardness.
- A measurement and observability layer on top — records every step, scores automatically on an eval set, catches regressions, and traces every task end to end.
The claim: not a tool that builds agents, but a tool that makes agents trustworthy.
This is what gets built over the six months (see docs/architecture.md):
┌──────────────────────────────┐
user ───────────▶│ control-plane (Java 21) │
│ Spring Boot │
│ · API surface & auth │
│ · task orchestration │
│ · persistence (JPA/Flyway) │
│ · task queue (Month 4) │
└───────┬──────────────┬───────┘
│ HTTP │ JDBC
▼ ▼
┌────────────────┐ ┌──────────────┐
│ agent-runtime │ │ PostgreSQL │
│ Python/FastAPI │──│ + pgvector │
│ · agent loop │ │ agent_task │
│ · tool calling │ │ agent_step │
│ · RAG │ └──────────────┘
│ · eval harness │
└────────────────┘
Why two services, why two languages? Python is the home of the LLM ecosystem; the agent runtime naturally lives there. The JVM is mature on the control, security, concurrency, and orchestration side. Leaving that split for later means a painful refactor — so they are separate from day one. Details: ADR-0001.
| Tool | Version | For |
|---|---|---|
| Docker + Compose | v2+ | One-command environment |
| Python | 3.11+ | agent-runtime |
| JDK | 21 | control-plane |
| Maven | 3.9+ | Java build |
| make | — | Command shortcuts |
| Month | Layer | Status |
|---|---|---|
| 1 | Two-service skeleton, data model, Docker | 🟡 in progress |
| 2 | Agent loop, tool calling, RAG | ⚪ planned |
| 3 | Eval harness and baseline ← the core of the project | ⚪ planned |
| 4 | Concurrency, queue, cache, cost tracking | ⚪ planned |
| 5 | Security, observability, CI/CD | ⚪ planned |
| 6 | Open-source launch | ⚪ planned |
Detailed roadmap: docs/roadmap.md
This section is filled with real numbers starting from Month 3. Nothing is written to this table before a baseline is recorded — an unmeasured improvement is not an improvement.
| Metric | Baseline | Current | Change |
|---|---|---|---|
| Task success rate | — | — | — |
| Tool selection accuracy | — | — | — |
| Average step count | — | — | — |
| Cost per task | — | — | — |
| p95 latency | — | — | — |
agentlens/
├── docs/ Architecture, ADRs, roadmap — the design and its rationale
├── wiki/ Team wiki: getting started, monthly plans, glossary, workflow
├── scripts/ GitHub setup tooling (repo, labels, protection, issues)
├── .github/ Issue/PR templates, labels, CODEOWNERS, workflow templates
└── (built during the program) agent-runtime/, control-plane/, docker-compose.yml, ...
The application code lives under agent-runtime/ and control-plane/ once you build them in Month 1 — see Wiki › Month 1.
- @canquesse — Maintainer
- @Simurg41 — Maintainer
Both maintainers work across the whole stack. Every month both learn the same topics; only the implementation is divided, and the two of them decide who takes what. Details: Wiki › Team and Roles.
Contributions are welcome. Before you start, read CONTRIBUTING.md and CODE_OF_CONDUCT.md.
For newcomers: issues labeled good first issue.
If you find a vulnerability, please do not open a public issue. Follow the guidance in SECURITY.md.