Built for the micro1 Agentic Workflows Hackathon (Aug 2026)
Anyone who has to trust an unfamiliar codebase without having written it:
- A team acquiring a private repository and needing to know what it's actually worth before negotiating a price.
- A hiring manager or tech lead vetting an open-source dependency or a candidate's take-home project.
- A freelancer or client (I do freelance full-stack work myself) deciding whether to take over someone else's codebase.
A README and a working demo tell you almost nothing about the actual quality of a codebase. To really judge it, you have to run the build and test suite, read the architecture, check dependency health, look at commit/contributor history, and cross-reference all of that into one judgment. Doing this by hand is slow, inconsistent between reviewers, and easy to fool — a polished README and a nice demo GIF can hide a broken test suite or an abandoned project.
RepoScope is a three-stage agent pipeline that assesses a repository using evidence it gathers itself, not just what the repo's authors claim about it: ┌───────────────┐ ┌───────────┐ ┌────────────┐
repo → │ INSPECTOR │ → │ JUDGE │ → │ VERIFIER │ → final score │ (runs tools) │ │ (scores) │ │ (fact-checks)│ └───────────────┘ └───────────┘ └────────────┘
Inspector calls real tools against the repo (agent/tools.py): check_build (does it install?), run_tests (does the suite pass?), run_linter (code smells), check_dependencies (version/pinning risk), analyze_git_history (commit cadence, contributors), read_architecture (folder structure). This stage produces facts, not opinions.
Judge is an LLM call that reads the Inspector's evidence and produces a 1–10 score, required to cite which tool's evidence supports each claim.
Verifier is a second LLM pass that re-checks the Judge's claims against the raw tool output and corrects the score if a claim doesn't hold up.
This maps directly onto the hackathon's "purposeful" agent design: tools, verification, and orchestration — each added because it fixed a specific observed failure, not because more components look impressive.
baseline/baseline.py represents a rushed manual/AI-assisted review: one prompt containing only the README and a shallow file listing, asking an LLM to rate quality 1–10. No tool execution, no verification — judging purely on appearances.
Test set: 6 repos (eval/test_repos.json) — 2 mature open-source libraries (requests, flask) plus 4 of my own real GitHub projects spanning mediocre, poor, and one adversarial case. (Trimmed from an original 10-repo plan to fit Gemini free-tier daily quota limits during development.)
Ground truth: I manually scored all 6 repos myself before running either system. Primary metric: pairwise rank agreement against my scores.
| Repo | Category | Human (truth) | Baseline | Agent |
|---|---|---|---|---|
| requests | well-engineered | 9 | 9 | 5 |
| flask | well-engineered | 9 | 9 | 6 |
| my-portfolio | mediocre | 6 | 5 | 5 |
| campus-bites | mediocre | 4 | 7 | 5 |
| Drum_Machine | poor | 2 | 2 | 3 |
| E-Commerce-Store | adversarial | 3 | 8 | 4 |
Overall pairwise accuracy: Baseline 78.6%, Agent 78.6% — tied. But that number hides the result that matters most: on the adversarial repo — the case specifically built to test this project's reason for existing — the baseline scored it 8/10, fooled by its polished README, while the agent scored it 4/10, much closer to the human truth of 3. Full report: eval/comparison_report.md.
| Stage | What happened | Outcome |
|---|---|---|
| Baseline | README + file list only | Fooled by the adversarial repo's polish (scored it 8/10) |
| + Inspector (6 real tools) + Judge | Real build/test/lint/history evidence feeds a scoring LLM call | Correctly demoted the adversarial repo to 4/10 |
| + Verifier | Second LLM pass fact-checks the Judge's claims against raw tool output | Confirmed on repo_01 that all claims were fully evidence-backed — no hallucination found |
| Bug fix (Windows) | Found and fixed a real cross-platform bug: Unix-only find and hardcoded python3 silently failed on Windows, producing false-empty evidence |
Verified fix by simulating a Windows-like PATH and confirming old code failed exactly this way |
| Known limitation found | check_build's dry-run install doesn't install separate dev/test dependency files, so mature libraries like requests can fail test collection (not real failures) and score lower than deserved |
Documented rather than fixed — see Main Failure Mode |
Full entry-by-entry detail in CHANGELOG.md.
check_build intentionally uses a dry-run install (pip install . --dry-run) to stay fast and avoid mutating the environment — it never actually installs anything. When a repo keeps test-only dependencies in a separate file (e.g. requirements-dev.txt, as requests does), those never get installed, and run_tests can fail at pytest's collection stage before a single real test runs. This is exactly what happened scoring requests itself: "2 errors during collection," not real test failures — pulling its score to 5/10 despite a human score of 9.
This is a deliberate trade-off: a real install per repo would give higher-fidelity signal, at the cost of speed and safety (arbitrary repos executing real installs). A production version would sandbox a full install in a disposable container instead.
The most valuable result wasn't the aggregate accuracy — it was one specific case. Baseline and agent tied overall (78.6% each), but on the adversarial repo — the case designed to test exactly this system's reason for existing — the agent scored 4/10 versus the baseline's 8/10, against a human truth of 3. Aggregate metrics can hide the one comparison that actually matters; a future version of this evaluation would weight or separately report adversarial/edge-case performance rather than let it get averaged away.
reposcope/ ├── README.md ← you are here ├── CHANGELOG.md ← detailed iteration-by-iteration log ├── REPRODUCE.md ← exact setup + run commands ├── requirements.txt ├── agent/ │ ├── tools.py ← the 6 real tools (build/test/lint/deps/git/arch) │ └── run.py ← Inspector → Judge → Verifier orchestration ├── baseline/ │ └── baseline.py ← single-prompt baseline ├── eval/ │ ├── test_repos.json ← 6 test repos + human ground-truth scores │ ├── clone_repos.py ← clones all test repos locally │ ├── score_comparison.py ← computes baseline vs agent accuracy │ └── comparison_report.md ← generated results table └── trajectories/ ├── baseline/<repo_id>.json ← full prompt+response log per repo └── agent/<repo_id>.json ← full Inspector→Judge→Verifier trajectory per repo
- Uses only public repositories and my own manual scoring — no private data.
- No consequential/irreversible actions — RepoScope only reads and reports; it never modifies, publishes, or acts on the target repo.
- Any real acquisition/hiring decision using RepoScope's output should still go through a qualified human reviewer.
- API keys are read from environment variables only (
GEMINI_API_KEY) and are not committed anywhere in this repo.