Skip to content

Real World Examples

Pengfei Hu edited this page Apr 26, 2026 · 3 revisions

Real-World Examples

Three reproducible scans against public production tool surfaces. Numbers below come from agents-shipgate v0.2.0. All sources are public; nothing in these scans makes a network call beyond fetching the spec file.


DigitalOcean public API

~600 endpoints — droplets, Kubernetes, apps, databases, SSH key management. Imagined as the tool surface for a cloud-ops agent with broad scopes.

Setup

mkdir do && cd do
curl -sLo openapi.yaml \
  https://raw.githubusercontent.com/digitalocean/openapi/main/specification/DigitalOcean-public.v2.yaml

shipgate.yaml:

version: "0.1"
project:
  name: digitalocean-api-as-agent-tools
  owner: platform-team
agent:
  name: digitalocean-ops-agent
  declared_purpose:
    - manage cloud infrastructure on DigitalOcean
    - create, update, and delete droplets and kubernetes clusters
  prohibited_actions:
    - delete production database clusters without approval
environment:
  target: production_like
tool_sources:
  - id: do_openapi
    type: openapi
    path: openapi.yaml
permissions:
  scopes: ["*"]
ci:
  mode: strict
  fail_on: [critical, high]

Run

agents-shipgate scan -c shipgate.yaml

Result

Tools Critical High Medium
591 142 659 609

Top findings. All 90 delete_v2_* operations + 27 post_v2_*_cancel|create|deploy + 21 put_v2_* are flagged as SHIP-POLICY-APPROVAL-MISSING. The wildcard scope fires SHIP-AUTH-MANIFEST-BROAD-SCOPE. Production target plus 591 tools fires SHIP-INVENTORY-TOOL-SURFACE-TOO-LARGE.

Representative finding (JSON):

{
  "id": "fp_…",
  "check_id": "SHIP-POLICY-APPROVAL-MISSING",
  "severity": "critical",
  "tool_name": "delete_v2_droplets_droplet_id",
  "evidence": {
    "risk_tags": ["destructive", "infrastructure_change", "write"],
    "policy_match": null
  },
  "source": {
    "type": "openapi",
    "ref": "openapi.yaml#/paths/~1v2~1droplets~1{droplet_id}/delete"
  },
  "recommendation": "Declare an approval policy for delete_v2_droplets_droplet_id or remove this tool from the release."
}

Why this matters. With a single retry on a transient error, an LLM holding this tool surface plus a wildcard token could plausibly delete a production cluster. Shipgate flags every irreversible action missing a policy entry — turning a 600-tool surface into a finite review list.

Tuning down the noise. The 3 remaining false-positive GETs (get_v2_*_destroy_with_associated_resources) are read-only status endpoints that contain the word destroy in their operationId. Suppress them with:

checks:
  ignore:
    - check_id: SHIP-POLICY-APPROVAL-MISSING
      tool: get_v2_droplets_droplet_id_destroy_with_associated_resources
      reason: GET-only status endpoint despite the destroy keyword in path

Twilio Messaging API

Real telecom API. The manifest deliberately claims a read-only purpose; Shipgate detects the contradiction with the spec's destructive operations.

Setup

mkdir twilio && cd twilio
curl -sLo twilio_messaging.yaml \
  https://raw.githubusercontent.com/twilio/twilio-oai/main/spec/yaml/twilio_messaging_v1.yaml

shipgate.yaml:

version: "0.1"
project:
  name: twilio-messaging-as-agent-tools
agent:
  name: customer-sms-agent
  declared_purpose:
    - look up customer messaging history
    - answer questions about message delivery status
  prohibited_actions:
    - send SMS to external customers without approval
    - modify message records
environment:
  target: production_like
tool_sources:
  - id: twilio_messaging
    type: openapi
    path: twilio_messaging.yaml
ci:
  mode: strict
  fail_on: [critical, high]

Result

Tools Critical High Medium
167 10 90 53

All 10 criticals are Delete* operations on the DELETE method:

DeleteAlphaSender              DeleteLinkshorteningMessagingService
DeleteChannelSender            DeletePhoneNumber
DeleteDestinationAlphaSender   DeleteService
DeleteDomainCertV4             DeleteShortCode
DeleteTollfreeVerification     DeleteUsAppToPerson

100% true-positive rate — every one is a real destructive operation.

Why this matters. A read-only service-account agent should never have these endpoints in its tool surface. Shipgate makes the contradiction explicit so the team merges either a narrower spec or an updated declared purpose — not both quietly diverging.


OpenAI Agents SDK · customer_service

The canonical openai-agents-python example, statically extracted from examples/customer_service/main.py — without importing or executing it.

The file under review

from agents import Agent, function_tool

@function_tool(
    name_override="faq_lookup_tool",
    description_override="Lookup frequently asked questions.",
)
async def faq_lookup_tool(question: str) -> str:
    ...

@function_tool
async def update_seat(
    context: RunContextWrapper[AirlineAgentContext],
    confirmation_number: str,
    new_seat: str,
) -> str:
    ...

Setup

git clone --depth 1 https://github.com/openai/openai-agents-python.git
cd openai-agents-python

Drop shipgate.yaml into the repo root:

version: "0.1"
project:
  name: openai-agents-sdk-customer-service
agent:
  name: airline-customer-service-agent
  sdk:
    type: openai_agents
    language: python
    entrypoint: examples/customer_service/main.py
  declared_purpose:
    - assist airline customers with seat changes and FAQs
environment:
  target: production
tool_sources:
  - id: customer_service_sdk
    type: openai_agents_sdk
    path: examples/customer_service/main.py
ci:
  mode: strict

Result

Tools Critical High Medium
2 0 2 2

Tool inventory (from report.json):

Name Source Risk tags Confidence
faq_lookup_tool sdk_function read_only medium
update_seat sdk_function write medium

Findings.

Severity Check Tool Why
high SHIP-INVENTORY-LOW-CONFIDENCE-PRODUCTION-SURFACE (agent) Production target plus medium-confidence SDK extraction
high SHIP-AUTH-MISSING-SCOPE update_seat Write tool with no declared scopes
medium SHIP-SCHEMA-FREEFORM-OUTPUT faq_lookup_tool Returns str
medium SHIP-SCHEMA-FREEFORM-OUTPUT update_seat Returns str

Why this matters. Shipgate identified update_seat as a write tool from its name alone — no HTTP method, no docstring keywords, just AST-extracted name plus the keyword classifier. The decorator's description_override was correctly read from kwargs (not falling back to the docstring). RunContextWrapper first-arg was correctly skipped as a context parameter.

The SHIP-INVENTORY-LOW-CONFIDENCE-PRODUCTION-SURFACE finding is the safety net: SDK static extraction is medium-confidence by design (decorators, dynamic factories, and runtime-built tools can hide), so production targets get an explicit nudge to declare tools through MCP/OpenAPI for higher-confidence inventory.


Reproducing locally

The full set of scans takes about 90 seconds:

pipx install agents-shipgate
mkdir -p ~/shipgate-tour && cd ~/shipgate-tour

# 1. DigitalOcean
mkdir do && (cd do && \
  curl -sLo openapi.yaml https://raw.githubusercontent.com/digitalocean/openapi/main/specification/DigitalOcean-public.v2.yaml && \
  cat > shipgate.yaml <<'EOF'
version: "0.1"
project: { name: digitalocean }
agent:   { name: do-ops, declared_purpose: [manage cloud infra] }
environment: { target: production_like }
tool_sources: [{ id: do, type: openapi, path: openapi.yaml }]
permissions: { scopes: ["*"] }
EOF
  agents-shipgate scan -c shipgate.yaml --ci-mode advisory)

# 2. Twilio
mkdir twilio && (cd twilio && \
  curl -sLo spec.yaml https://raw.githubusercontent.com/twilio/twilio-oai/main/spec/yaml/twilio_messaging_v1.yaml && \
  cat > shipgate.yaml <<'EOF'
version: "0.1"
project: { name: twilio }
agent:
  name: customer-sms-agent
  declared_purpose: [look up customer messaging history]
environment: { target: production_like }
tool_sources: [{ id: tw, type: openapi, path: spec.yaml }]
EOF
  agents-shipgate scan -c shipgate.yaml --ci-mode advisory)

# 3. OpenAI Agents SDK
git clone --depth 1 https://github.com/openai/openai-agents-python.git sdk
(cd sdk && \
  cat > shipgate.yaml <<'EOF'
version: "0.1"
project: { name: openai-agents-customer-service }
agent:
  name: airline-cs
  declared_purpose: [assist airline customers]
  sdk: { type: openai_agents, language: python, entrypoint: examples/customer_service/main.py }
environment: { target: production }
tool_sources:
  - { id: sdk, type: openai_agents_sdk, path: examples/customer_service/main.py }
EOF
  agents-shipgate scan -c shipgate.yaml --ci-mode advisory)

Each run writes agents-shipgate-reports/report.md and report.json next to its shipgate.yaml.


What these examples demonstrate

Property Where it shows
Scales to ~600 endpoints DigitalOcean: 591 tools, ~290 ms wall-clock
Catches purpose contradictions Twilio: read-only purpose vs Delete* operations
Static SDK extraction works without import OpenAI Agents SDK: AST-only, no execution
Decorator metadata is honored description_override on faq_lookup_tool
Production target adds safety net SHIP-INVENTORY-LOW-CONFIDENCE-PRODUCTION-SURFACE on the SDK example
Path-traversal containment A malicious path: ../../etc/passwd is rejected with Input path resolves outside manifest directory
Plugin trust boundary --no-plugins and loaded_plugins[] provenance work end-to-end

These behaviors are pinned by the test suite and reproducible from the public sources above.

Clone this wiki locally