Repository navigation
Tool Surface Study 2026
Prepublication research browser. The current public artifacts contain the preregistered catalog, fixed repository snapshots, historical-corpus lineage, and workflow status. They do not contain completed scan findings, project rankings, or runtime-safety claims.
Open the no-login study browser →
Sandbox controls can. Evals measure does. Agents Shipgate reviews should vs. can before merge.
Sandboxes constrain runtime capability. Evals observe behavior in tested scenarios. Agents Shipgate compares the static tool authority introduced by a change with the purpose, scopes, and controls reviewers intended to approve.
Static only: Agents Shipgate does not execute the agent, replace runtime sandboxing, or prove runtime behavior.
The study asks what static tool-surface risk signals and review or coverage gaps are visible across fixed snapshots of prominent open-source Agent projects. It is a deterministic static evidence study—not a vulnerability scan or a runtime-safety ranking.
The primary census contains ten deployable Agent applications at fixed commits:
- General / personal: OpenClaw, AutoGPT Classic, OpenManus
- Coding: Gemini CLI, Codex, OpenHands, Cline
- Web / research: browser-use, Deep Agents, Open Deep Research
Four repositories form a separate adapter-calibration layer:
- OpenAI Agents SDK
- Google ADK samples
- CrewAI
- n8n AI starter kit
Magentic-UI is the only preregistered alternate and may replace a primary sample only if license or archive status invalidates that sample before results are viewed. Static unsupported status remains a result; it is not a replacement condition.
| Gate | Current public state |
|---|---|
| Registered repositories | 14 |
| Fixed commit SHAs | 14 |
| Automatically detected source candidates | 6 |
| Human-approved source maps | 0 |
| Publishable results | 0 |
These numbers describe workflow state, not project safety. Not found does not mean absent. Unsupported and failed cases remain visible and stay in the final denominator.
- immutable repository and historical-corpus locks
- pinned snapshot metadata and candidate scan scopes
- JSON, JSONL, and CSV prepublication status projections
- a SHA-256 artifact manifest
- explicit status for all preregistered samples
The public status data is an inspection projection. Raw reports, SARIF, complete Capability Locks, and undisclosed evidence are not part of the prepublication bundle.
The report uses static risk signal, coverage gap, and review gap. It does not publish a composite risk score, project leaderboard, “vulnerability rate,” “safety rate,” or “most dangerous Agent” claim.
A static passed decision does not mean runtime safety.
insufficient_evidence does not mean a project is unsafe. Purpose alignment is
reported as confirmed_mismatch, aligned_on_available_evidence,
needs_context, or unassessable—never safe.
- Two reviewers approve each deployable subproject, source map, and purpose source.
- A pinned, qualified engine runs the primary and
--no-heuristicsscans twice. - Two independent reviewers blind-label purpose and alignment; a third human adjudicates disagreements.
- Named findings receive a ten-business-day maintainer fact-check window.
- Public artifacts pass schema, reproducibility, privacy, disclosure, and hash gates.
The reproduction command shown on each browser card is intended to run from an Agents Shipgate repository checkout. The downloadable catalog is the public inspection copy; reproduction remains gated until the corresponding source map and engine decisions are approved.
The existing merged-PR corpus contains 361 ingestion rows normalized to 321 unique PR URLs. The unique-PR count is the analysis denominator; 361 is retained only as data lineage. The same frozen engine will replay the corpus, with blind human review of engine-engaged PRs and 15 seeded trigger-skip negative controls.
Five cases are preregistered for explanation, not new prevalence estimates: Stripe AI #232, CrewAI examples #169, ADK samples #1975, OpenAI Agents Python #3461, and Goose #9637.
The browser is intentionally noindex and excluded from the site sitemap until
the publication gates are complete.
Agents Shipgate · Apache-2.0 · maintained by Three Moons Lab · 2026 tool-surface study · Report a false positive
Getting started
Reference
Workflows
Research
Extending
Project