llm-dev-workbench

A structured knowledge base for LLM-assisted software development.

When an AI coding agent works on your project, it sees the code — but not the context around it: architecture decisions, deployment topology, project interdependencies, current priorities. Without this context, the agent makes locally correct but globally wrong decisions.

llm-dev-workbench is a pattern (and a set of tools) for maintaining a centralized knowledge base that plugs into your IDE as a second workspace root, giving the agent persistent access to cross-project context.

The Problem

LLM agents trust what they see. Stale documentation is worse than no documentation — it actively poisons the agent's decisions. The agent reads an outdated project registry, treats it as current, and builds on information that no longer reflects reality.

With dozens of projects evolving in parallel, every change potentially invalidates something in the knowledge base. This isn't a discipline problem — it's a scale problem. Hundreds of documents, dozens of projects, a continuous stream of changes. Manual maintenance doesn't scale. You need automated detection of staleness and behavioral rules that prevent it from accumulating.

Directory Structure

A workbench follows a simple convention:

workbench/
├── projects/           # per-project documentation
│   ├── my-api/
│   │   ├── README.md
│   │   ├── backlog.md
│   │   └── architecture.md
│   └── my-frontend/
│       ├── README.md
│       └── backlog.md
├── domains/            # cross-project knowledge by topic
│   ├── infrastructure/
│   │   ├── deployment_topology.md
│   │   └── monitoring.md
│   └── data-platform/
│       ├── schema_registry.md
│       └── data_flow.md
├── workspaces/         # IDE workspace files (.code-workspace)
│   ├── my-api.code-workspace
│   └── my-frontend.code-workspace
├── tools/              # validation and maintenance scripts
│   ├── validate.sh
│   ├── epistemic_health.sh
│   ├── collect_backlog.sh
│   └── sync_repos.sh
└── archive/            # outdated but preserved documents

Key principles:

• projects/<name>/ mirrors your actual project repositories. Each subfolder contains documentation specific to one project: backlogs, architecture notes, investigation logs.
• domains/<name>/ holds knowledge that spans multiple projects: infrastructure topology, shared libraries documentation, hardware specs.
• workspaces/ contains .code-workspace files that pair a project repository with the workbench, so the agent always has access to both.
• archive/ preserves outdated documents without mixing them with active ones. Unlike git history, archived documents remain visible to the agent but are clearly marked as superseded.

Document Lifecycle

Every document carries metadata in its header:

# Deployment Topology

- **Status:** active
- **Last updated:** 2026-03-15

Four statuses:

Status	Meaning	Freshness expectation
active	Actively maintained, reflects current state	Updated regularly
reference	Stable reference material	Updated on change
draft	Work in progress	N/A
archived	Superseded, kept for context	Never

The agent doesn't distinguish a fresh plan from a three-month-old artifact if both sit in the same folder without metadata. Explicit status and date give the agent a signal it otherwise lacks.

Tools

validate.sh

Structural integrity checker for CI or manual runs. Verifies:

• Internal markdown links resolve to existing files
• Key documents are not stale (configurable threshold, default 60 days)
• Every project directory has a corresponding workspace file
• Workspace files reference existing directories

./tools/validate.sh

# Custom freshness threshold
STALE_THRESHOLD_DAYS=30 ./tools/validate.sh

# If workspaces use a canonical base path
BASE_DIR=/home/user/dev ./tools/validate.sh

Example output:

=== Workbench validation ===
  Root: /home/user/workbench

[markdown links]
  WARN:  broken link in domains/infra/topology.md -> decisions/adr_007.md
  INFO:  1 broken link(s) out of 142 checked

[document freshness — threshold 60d]
  WARN:  stale (87d): projects/my-api/hardware_notes.md
  INFO:  1 document(s) exceed freshness threshold

[project coverage]
  OK:    all projects have workspace files

=== Result ===
  Errors: 0 | Warnings: 2 | Info: 2

Passed with 2 warning(s).

epistemic_health.sh

Visual health report across four axes: coverage, staleness, broken links, and domain balance. Designed for manual review rather than CI.

./tools/epistemic_health.sh           # full report
./tools/epistemic_health.sh --summary # summary only
STALE_DAYS=30 ./tools/epistemic_health.sh

collect_backlog.sh

Aggregates TODO items (markdown checkboxes) from all documents into a single view.

./tools/collect_backlog.sh                     # open items
./tools/collect_backlog.sh --with-done         # include completed
./tools/collect_backlog.sh --filter "deploy"   # filter by keyword

sync_repos.sh

Discovers git repositories under a base directory and syncs them with remotes. Filters by recency to skip dormant repos, handles dirty working trees and diverged branches gracefully.

./tools/sync_repos.sh                     # sync repos active in last 30 days
./tools/sync_repos.sh --days 7            # last week only
./tools/sync_repos.sh --all --dry-run     # preview full sync plan
DEV_DIR=/home/user/code ./tools/sync_repos.sh

Getting Started

1. Create a workbench directory and initialize it as a git repo:

   mkdir -p ~/workbench/{projects,domains,workspaces,tools}
   cd ~/workbench && git init

2. Copy the tools:

   cp -r path/to/llm-dev-workbench/tools/ ~/workbench/tools/
   chmod +x ~/workbench/tools/*.sh

3. For each project, create a workspace file that pairs the project with the workbench:

   {
     "folders": [
       { "path": "/home/user/dev/my-project" },
       { "path": "/home/user/workbench" }
     ]
   }

   Save as workspaces/my-project.code-workspace and open it in your IDE.

4. Add project documentation:

   mkdir ~/workbench/projects/my-project
   # Add README.md, backlog.md, architecture notes, etc.

5. Run validation:

   ./tools/validate.sh

Behavioral Rules

Tools detect problems but don't solve them. What closes the loop is discipline encoded in agent instructions (e.g., Cursor rules, AGENTS.md, or system prompts):

1. Post-phase updates: After completing a work phase, the agent updates related workbench documents — statuses, metrics, backlogs — and surfaces a list of remaining tasks and discovered issues.

2. Continuation prompts: When the context window approaches its limit, the agent produces a structured handoff: what was done, current state, next tasks. The new session starts where the previous one stopped, not from scratch.

3. Same-session plan updates: Plans are updated in the same session where the change happens, not "later." A stale plan is poisoned context disguised as progress.

License

MIT