Repository navigation
Real World Examples
Three reproducible scans against public production tool surfaces. Numbers below come from agents-shipgate v0.2.0. All sources are public; nothing in these scans makes a network call beyond fetching the spec file.
~600 endpoints — droplets, Kubernetes, apps, databases, SSH key management. Imagined as the tool surface for a cloud-ops agent with broad scopes.
mkdir do && cd do
curl -sLo openapi.yaml \
https://raw.githubusercontent.com/digitalocean/openapi/main/specification/DigitalOcean-public.v2.yamlshipgate.yaml:
version: "0.1"
project:
name: digitalocean-api-as-agent-tools
owner: platform-team
agent:
name: digitalocean-ops-agent
declared_purpose:
- manage cloud infrastructure on DigitalOcean
- create, update, and delete droplets and kubernetes clusters
prohibited_actions:
- delete production database clusters without approval
environment:
target: production_like
tool_sources:
- id: do_openapi
type: openapi
path: openapi.yaml
permissions:
scopes: ["*"]
ci:
mode: strict
fail_on: [critical, high]agents-shipgate scan -c shipgate.yaml| Tools | Critical | High | Medium |
|---|---|---|---|
| 591 | 142 | 659 | 609 |
Top findings. All 90 delete_v2_* operations + 27 post_v2_*_cancel|create|deploy + 21 put_v2_* are flagged as SHIP-POLICY-APPROVAL-MISSING. The wildcard scope fires SHIP-AUTH-MANIFEST-BROAD-SCOPE. Production target plus 591 tools fires SHIP-INVENTORY-TOOL-SURFACE-TOO-LARGE.
Representative finding (JSON):
{
"id": "fp_…",
"check_id": "SHIP-POLICY-APPROVAL-MISSING",
"severity": "critical",
"tool_name": "delete_v2_droplets_droplet_id",
"evidence": {
"risk_tags": ["destructive", "infrastructure_change", "write"],
"policy_match": null
},
"source": {
"type": "openapi",
"ref": "openapi.yaml#/paths/~1v2~1droplets~1{droplet_id}/delete"
},
"recommendation": "Declare an approval policy for delete_v2_droplets_droplet_id or remove this tool from the release."
}Why this matters. With a single retry on a transient error, an LLM holding this tool surface plus a wildcard token could plausibly delete a production cluster. Shipgate flags every irreversible action missing a policy entry — turning a 600-tool surface into a finite review list.
Tuning down the noise. The 3 remaining false-positive GETs (get_v2_*_destroy_with_associated_resources) are read-only status endpoints that contain the word destroy in their operationId. Suppress them with:
checks:
ignore:
- check_id: SHIP-POLICY-APPROVAL-MISSING
tool: get_v2_droplets_droplet_id_destroy_with_associated_resources
reason: GET-only status endpoint despite the destroy keyword in pathReal telecom API. The manifest deliberately claims a read-only purpose; Shipgate detects the contradiction with the spec's destructive operations.
mkdir twilio && cd twilio
curl -sLo twilio_messaging.yaml \
https://raw.githubusercontent.com/twilio/twilio-oai/main/spec/yaml/twilio_messaging_v1.yamlshipgate.yaml:
version: "0.1"
project:
name: twilio-messaging-as-agent-tools
agent:
name: customer-sms-agent
declared_purpose:
- look up customer messaging history
- answer questions about message delivery status
prohibited_actions:
- send SMS to external customers without approval
- modify message records
environment:
target: production_like
tool_sources:
- id: twilio_messaging
type: openapi
path: twilio_messaging.yaml
ci:
mode: strict
fail_on: [critical, high]| Tools | Critical | High | Medium |
|---|---|---|---|
| 167 | 10 | 90 | 53 |
All 10 criticals are Delete* operations on the DELETE method:
DeleteAlphaSender DeleteLinkshorteningMessagingService
DeleteChannelSender DeletePhoneNumber
DeleteDestinationAlphaSender DeleteService
DeleteDomainCertV4 DeleteShortCode
DeleteTollfreeVerification DeleteUsAppToPerson
100% true-positive rate — every one is a real destructive operation.
Why this matters. A read-only service-account agent should never have these endpoints in its tool surface. Shipgate makes the contradiction explicit so the team merges either a narrower spec or an updated declared purpose — not both quietly diverging.
The canonical openai-agents-python example, statically extracted from examples/customer_service/main.py — without importing or executing it.
from agents import Agent, function_tool
@function_tool(
name_override="faq_lookup_tool",
description_override="Lookup frequently asked questions.",
)
async def faq_lookup_tool(question: str) -> str:
...
@function_tool
async def update_seat(
context: RunContextWrapper[AirlineAgentContext],
confirmation_number: str,
new_seat: str,
) -> str:
...git clone --depth 1 https://github.com/openai/openai-agents-python.git
cd openai-agents-pythonDrop shipgate.yaml into the repo root:
version: "0.1"
project:
name: openai-agents-sdk-customer-service
agent:
name: airline-customer-service-agent
sdk:
type: openai_agents
language: python
entrypoint: examples/customer_service/main.py
declared_purpose:
- assist airline customers with seat changes and FAQs
environment:
target: production
tool_sources:
- id: customer_service_sdk
type: openai_agents_sdk
path: examples/customer_service/main.py
ci:
mode: strict| Tools | Critical | High | Medium |
|---|---|---|---|
| 2 | 0 | 2 | 2 |
Tool inventory (from report.json):
| Name | Source | Risk tags | Confidence |
|---|---|---|---|
faq_lookup_tool |
sdk_function | read_only |
medium |
update_seat |
sdk_function | write |
medium |
Findings.
| Severity | Check | Tool | Why |
|---|---|---|---|
| high | SHIP-INVENTORY-LOW-CONFIDENCE-PRODUCTION-SURFACE |
(agent) | Production target plus medium-confidence SDK extraction |
| high | SHIP-AUTH-MISSING-SCOPE |
update_seat |
Write tool with no declared scopes |
| medium | SHIP-SCHEMA-FREEFORM-OUTPUT |
faq_lookup_tool |
Returns str
|
| medium | SHIP-SCHEMA-FREEFORM-OUTPUT |
update_seat |
Returns str
|
Why this matters. Shipgate identified update_seat as a write tool from its name alone — no HTTP method, no docstring keywords, just AST-extracted name plus the keyword classifier. The decorator's description_override was correctly read from kwargs (not falling back to the docstring). RunContextWrapper first-arg was correctly skipped as a context parameter.
The SHIP-INVENTORY-LOW-CONFIDENCE-PRODUCTION-SURFACE finding is the safety net: SDK static extraction is medium-confidence by design (decorators, dynamic factories, and runtime-built tools can hide), so production targets get an explicit nudge to declare tools through MCP/OpenAPI for higher-confidence inventory.
The full set of scans takes about 90 seconds:
pipx install agents-shipgate
mkdir -p ~/shipgate-tour && cd ~/shipgate-tour
# 1. DigitalOcean
mkdir do && (cd do && \
curl -sLo openapi.yaml https://raw.githubusercontent.com/digitalocean/openapi/main/specification/DigitalOcean-public.v2.yaml && \
cat > shipgate.yaml <<'EOF'
version: "0.1"
project: { name: digitalocean }
agent: { name: do-ops, declared_purpose: [manage cloud infra] }
environment: { target: production_like }
tool_sources: [{ id: do, type: openapi, path: openapi.yaml }]
permissions: { scopes: ["*"] }
EOF
agents-shipgate scan -c shipgate.yaml --ci-mode advisory)
# 2. Twilio
mkdir twilio && (cd twilio && \
curl -sLo spec.yaml https://raw.githubusercontent.com/twilio/twilio-oai/main/spec/yaml/twilio_messaging_v1.yaml && \
cat > shipgate.yaml <<'EOF'
version: "0.1"
project: { name: twilio }
agent:
name: customer-sms-agent
declared_purpose: [look up customer messaging history]
environment: { target: production_like }
tool_sources: [{ id: tw, type: openapi, path: spec.yaml }]
EOF
agents-shipgate scan -c shipgate.yaml --ci-mode advisory)
# 3. OpenAI Agents SDK
git clone --depth 1 https://github.com/openai/openai-agents-python.git sdk
(cd sdk && \
cat > shipgate.yaml <<'EOF'
version: "0.1"
project: { name: openai-agents-customer-service }
agent:
name: airline-cs
declared_purpose: [assist airline customers]
sdk: { type: openai_agents, language: python, entrypoint: examples/customer_service/main.py }
environment: { target: production }
tool_sources:
- { id: sdk, type: openai_agents_sdk, path: examples/customer_service/main.py }
EOF
agents-shipgate scan -c shipgate.yaml --ci-mode advisory)Each run writes agents-shipgate-reports/report.md and report.json next to its shipgate.yaml.
| Property | Where it shows |
|---|---|
| Scales to ~600 endpoints | DigitalOcean: 591 tools, ~290 ms wall-clock |
| Catches purpose contradictions | Twilio: read-only purpose vs Delete* operations |
| Static SDK extraction works without import | OpenAI Agents SDK: AST-only, no execution |
| Decorator metadata is honored |
description_override on faq_lookup_tool
|
| Production target adds safety net |
SHIP-INVENTORY-LOW-CONFIDENCE-PRODUCTION-SURFACE on the SDK example |
| Path-traversal containment | A malicious path: ../../etc/passwd is rejected with Input path resolves outside manifest directory
|
| Plugin trust boundary |
--no-plugins and loaded_plugins[] provenance work end-to-end |
These behaviors are pinned by the test suite and reproducible from the public sources above.
Agents Shipgate · Apache-2.0 · maintained by Three Moons Lab · 2026 tool-surface study · Report a false positive
Getting started
Reference
Workflows
Research
Extending
Project