spec: Define API pattern investigation infrastructure — systematic process for new transport/protocol types

[mickdarling]

Summary

Define the infrastructure and methodology for investigating new API patterns and producing the transport plugins, protocol plugins, interrogation modules, and generator extensions needed to support them in MCPAQL. This is the process that produces new categories of adapters, not individual adapters.

Context

The native-applescript work (spec #227-233) demonstrated the full pattern investigation lifecycle:

1. API pattern research — How does AppleScript work? What's the execution model? What types exist?
2. Transport design — osascript subprocess, parameterized templates, JXA vs native AppleScript
3. maps_to syntax — A new mapping format for non-HTTP operations
4. Serialization contract — AppleScript types ↔ JSON conversion rules
5. Security analysis — Injection vectors specific to this pattern (script injection vs SQL injection vs HTTP injection)
6. Interrogation pathway — .sdef XML parsing as the discovery source
7. Generator extension — New code generation path for native transport
8. Spec authoring — Normative text for all of the above

This same process will repeat for every new API pattern MCPAQL encounters. Without infrastructure, each one is ad-hoc and the quality/thoroughness varies.

Known Upcoming API Patterns

Pattern	Transport	Protocol	Discovery Source	Security Model	Estimated Complexity
GraphQL	HTTP	graphql	Introspection query	Standard HTTP + query injection	Medium
gRPC	HTTP/2	grpc/protobuf	Reflection API / .proto files	mTLS, binary serialization	High
WebSocket	websocket	varies	Handshake + message schema	Connection hijacking, message injection	Medium
Database	tcp/socket	sql/nosql	Schema introspection (INFORMATION_SCHEMA, etc.)	SQL injection, credential handling	High
Serial/USB	serial	varies	Device descriptor / protocol docs	Physical access, firmware interaction	High
Cloud SDKs	HTTP	rest/custom	SDK type definitions, OpenAPI specs	OAuth, IAM, credential chains	Medium
Shell/CLI	subprocess	stdio	--help parsing, man pages, completion scripts	Command injection, env leakage	Medium
IPC/D-Bus	socket	dbus	Introspection XML	Privilege escalation	Medium
Browser automation	websocket	CDP/BiDi	Protocol spec	XSS, navigation hijacking	High
Email protocols	tcp	IMAP/SMTP	CAPABILITY response	Credential handling, injection	Medium

Proposed Infrastructure

Pattern Investigation Template

A structured research template that every new pattern investigation follows:

# Pattern Investigation: {Pattern Name}

## 1. API Surface Analysis
- What is the communication model? (request/response, streaming, pub/sub, RPC)
- What transport does it use? (HTTP, TCP, subprocess, IPC, serial)
- What serialization? (JSON, protobuf, XML, binary, text)
- What authentication? (bearer, mTLS, API key, session, none)
- Is there a machine-readable API description? (OpenAPI, .proto, WSDL, .sdef, introspection)

## 2. CRUDE Mapping Analysis
- How do operations naturally map to Create/Read/Update/Delete/Execute?
- Are there operations that don't fit? How should they be classified?
- What's the idempotency model?
- What's reversible vs destructive?

## 3. Transport Plugin Design
- Does an existing transport work or do we need a new one?
- What's the connection lifecycle? (per-request, persistent, pooled)
- How are errors reported? (status codes, exceptions, error objects)
- What's the timeout/retry model?

## 4. Protocol Plugin Design
- How does maps_to work for this pattern?
- How are parameters bound? (path, query, body, headers, positional)
- How are responses parsed?
- Are there pagination/streaming considerations?

## 5. Serialization Analysis
- What types exist in this system?
- How do they map to/from JSON?
- Are there ambiguous or lossy conversions?
- Are there binary types that need special handling?

## 6. Security Analysis
- What are the injection vectors specific to this pattern?
- What sanitization rules are needed?
- What's the credential handling model?
- What operations are irreversible/dangerous?
- What should be auto-denied?

## 7. Discovery/Interrogation Design
- What's the machine-readable source? (spec file, introspection API, SDK types)
- How do we extract operations, parameters, types?
- How do we classify operations into CRUDE endpoints?
- What confidence level can we achieve automatically vs requiring human review?

## 8. Generator Extension
- What does generated server code look like for this pattern?
- What runtime dependencies are needed?
- What's different from the HTTP generation path?

## 9. Prototype Validation
- Build minimal working prototype
- Test against real target
- Document what worked, what didn't, what the spec should say

## 10. Spec Authoring
- Transport plugin spec
- Protocol plugin spec (if new)
- Serialization contract (if non-JSON)
- Security requirements
- Interrogation format

Pattern Investigation Tooling

Tool	Purpose	Status
Investigation template	Structured research checklist	Proposed above
API surface analyzer	Automated classification of API characteristics	Not yet built
Security model generator	Produce sanitization rules from pattern analysis	Not yet built
Prototype scaffolder	Generate a skeleton transport plugin for rapid prototyping	Not yet built
Spec generator	Produce draft normative text from investigation results	Not yet built
Pattern registry	Track which patterns are supported, in progress, or planned	Not yet built

DollhouseMCP Agent Support

The applescript-adapter-engineer agent was purpose-built for one pattern. The infrastructure should support:

• A pattern investigator agent that can research a new API pattern and produce the investigation template
• A transport engineer agent that can implement a transport plugin from investigation results
• A spec author agent that can draft normative text from implementation experience
• Composable via ensembles for end-to-end pattern onboarding

Relationship to #236

Issue #236 (adapter onboarding pipeline) handles the workflow for creating individual adapters within an already-supported pattern. This issue handles the workflow for adding support for entirely new patterns. The pipeline in #236 consumes the outputs of this infrastructure.

This issue (#237): "How do we support GraphQL?"
  → Produces: graphql transport plugin, graphql protocol plugin, 
    graphql interrogation module, generator extension

Issue #236: "How do we create a GitHub GraphQL adapter?"
  → Consumes: the graphql infrastructure from above
  → Produces: a specific adapter for GitHub's GraphQL API

Acceptance Criteria

• Pattern investigation template defined and documented
• Pattern registry format specified (tracking supported/in-progress/planned patterns)
• Relationship between pattern investigation and adapter onboarding (spec: Define adapter onboarding pipeline — end-to-end workflow for new API targets #236) documented
• Tooling roadmap for pattern-level automation
• At least 3 patterns assessed using the template (AppleScript retrospective, GraphQL prospective, gRPC prospective)

References

• Native AppleScript investigation: spec spec: Define native-applescript transport plugin specification #227-233
• Adapter onboarding pipeline: spec spec: Define adapter onboarding pipeline — end-to-end workflow for new API targets #236
• Plugin contracts: docs/plugin-contracts.md
• Deferred plugins table: plugin-contracts.md Section 1.3

[mickdarling]

Pattern Investigation Retrospective: native-applescript (2026-04-04)

The native-applescript transport is the first completed pattern investigation. Documenting what the process actually looked like, as input for the investigation infrastructure design.

What We Actually Did (chronological)

1. API Surface Research (~15 min) — Tested AppleScript access via osascript commands. Discovered Apple Mail exposes accounts, mailboxes, messages with full metadata + body via AppleScript. Confirmed no FDA needed (scripting bridge bypasses it).

2. Spec Landscape Review (~20 min) — Read plugin-contracts.md, adapter element-type.md, discovery-bundle.md, generator.md. Confirmed native-* was listed as deferred but architecturally planned. Found the maps_to reinterpretation was the key design decision.

3. Issue Creation (~30 min) — Filed 7 spec issues (spec: Define native-applescript transport plugin specification #227-233) covering transport plugin, interrogation, sanitization, serialization, adapter, prototype, and generic pattern. Also 4 implementation issues across repos.

4. Prototype Implementation (~18 min agent time) — Background agent built transport plugin, sdef parser, generator extension, and adapter schema across 4 repos. 115 tests.

5. Review Cycle (~5 rounds, ~4 hours wall clock) — Claude bot reviewed, we fixed, re-reviewed. Discovered real bugs (regex escape in template literals, dead code, injection vectors, runtime bombs from optional params). Final state: 94 + 20 + 16 tests.

6. Pipeline Execution (~10 seconds) — Ran interrogator → schema builder → generator → npm install → server starts. End-to-end worked first try.

What the Investigation Template Should Capture

Based on this experience, the pattern investigation template (proposed in this issue) should include:

• Transport semantics: How does execution work? (subprocess, IPC, network) What's the lifecycle?
• maps_to design: How do CRUDE operations map to this API's native concepts? This was the hardest design decision.
• Serialization boundary: What types exist? How do they round-trip through JSON? Where is fidelity lost?
• Injection model: What are the unique injection vectors? (AppleScript injection ≠ SQL injection ≠ HTTP injection). Each pattern has its own threat model.
• Discovery source: What's the machine-readable API description? (.sdef, OpenAPI, .proto, introspection query). How complete is it?
• Classification confidence: How well do operations auto-classify into CRUDE? For sdef: commands map well, properties map well, but the 304-operation surface needs aggressive pruning.

Key Learnings for Future Patterns

1. The prototype-first approach was correct — Spec issues were filed first but the implementation informed the spec, not the other way around. Code-first, spec-second.
2. Security review caught real bugs — The regex escape issue ([\w] → [w] in template literals) was a genuine security vulnerability that only surfaced during adversarial testing. Every pattern needs security-focused review.
3. The overrides step is critical — 304 raw operations is unusable. The pipeline needs a guided curation step.
4. Template literal escaping is a landmine — Any pattern that generates code (not just native-applescript) will hit this. The generator needs a general solution for embedding regex/special chars in template literals.