diff --git a/README.md b/README.md index cbc83a6..70aac11 100644 --- a/README.md +++ b/README.md @@ -561,8 +561,10 @@ Prompt files use YAML front matter with these fields: | `fallback_models` | `string[]` | Fallback model list | | `reasoning` | `object` | `{ effort, budget_tokens }` | | `sampling` | `object` | `{ temperature, top_p, frequency_penalty, presence_penalty, stop, max_output_tokens }` | -| `response` | `object` | `{ format, stream }` | +| `response` | `object` | `{ format, stream, schema, schema_name, schema_strict }` | +| `cache` | `object` | Provider-specific cache controls (`openai`, `anthropic`, `gemini`/`google`) | | `tools` | `array` | Tool references (string names or inline definitions) | +| `provider_options` | `object` | Provider-specific non-portable options (`anthropic`, `gemini`) | | `mcp` | `object` | MCP server references | | `context` | `object` | `{ inputs, history }` — declare expected variables, with optional per-input `max_size`, `trim`, structured or literal `allow_regex`/`deny_regex`, and built-in `non_empty` / `reject_secrets` validators | | `includes` | `string[]` | Paths to included prompt files | diff --git a/SKILL.md b/SKILL.md index 7b27351..d0ff931 100644 --- a/SKILL.md +++ b/SKILL.md @@ -8,7 +8,7 @@ description: Guidance for creating and editing promptopskit prompt files, defaul This project uses **promptopskit** to manage LLM prompts as code. Prompts live in markdown files with YAML front matter, are validated against a schema, and render into provider-specific request bodies (OpenAI, Anthropic, -Gemini, OpenRouter). Follow these instructions when creating or editing prompts. +Gemini, OpenRouter, and OpenAI Responses). Follow these instructions when creating or editing prompts. --- @@ -58,13 +58,15 @@ the fields required by that specific file: | `id` | string | **yes** | Unique identifier for the prompt | | `schema_version` | number | yes | Always `1` | | `description` | string | no | Human-readable description | -| `provider` | enum | no | `openai`, `anthropic`, `google`, `gemini`, `openrouter`, or `any` | +| `provider` | enum | no | `openai`, `openai-responses`, `anthropic`, `google`, `gemini`, `openrouter`, or `any` | | `model` | string | no | Model identifier (e.g. `gpt-5.4`, `claude-sonnet-4-20250514`) | | `fallback_models` | string[] | no | Ordered fallback model list | | `reasoning` | object | no | `{ effort: low|medium|high, budget_tokens: number }` | | `sampling` | object | no | `{ temperature, top_p, frequency_penalty, presence_penalty, stop, max_output_tokens }` | -| `response` | object | no | `{ format: text|json|markdown, stream: boolean }` | +| `response` | object | no | `{ format: text|json|markdown, stream: boolean, schema?: object, schema_name?: string, schema_strict?: boolean }` | +| `cache` | object | no | Provider-specific cache controls (`openai`, `anthropic`, `gemini`/`google`) | | `tools` | array | no | Tool names (strings) or inline definitions with `{ name, description, input_schema }` | +| `provider_options` | object | no | Provider-specific advanced options (`anthropic`, `gemini`) | | `mcp` | object | no | `{ servers: [string | { name, config }] }` | | `context.inputs` | `Array` | no | Declared variable names used in templates, with optional size budgets and runtime hardening controls | | `context.history` | object | no | `{ max_items: number }` | @@ -182,6 +184,7 @@ prompts/ Supported default fields: - `provider` (front matter) — default provider for the folder - `model` (front matter) — default model for the folder +- `cache` (front matter) — default provider-specific cache hints - `metadata` (front matter) — merged with prompt-local metadata - `# System instructions` (body section) — used when the prompt has none @@ -229,7 +232,7 @@ tiers: ``` Overridable fields: `model`, `fallback_models`, `reasoning`, `sampling`, -`response`, `tools`. +`response`, `cache`, `tools`, `provider_options`. Override application order: **base → environment → tier → runtime**. diff --git a/docs/index.md b/docs/index.md index d979411..a0543c4 100644 --- a/docs/index.md +++ b/docs/index.md @@ -17,6 +17,7 @@ Open-source developer toolkit for managing prompts, system instructions, tools, - [CLI](./cli.md) — Command-line interface: init, validate, compile, render, inspect, skill - [API Reference](./api-reference.md) — TypeScript API: `createPromptOpsKit`, `renderPrompt`, standalone functions - [Schema](./schema.md) — Full YAML front matter schema reference +- [Vendor Schema Gap Analysis](./vendor-schema-gap-analysis.md) — Snapshot comparison against published OpenAI, Anthropic, Gemini, and OpenRouter schema capabilities - [Testing](./testing.md) — Test helpers, mock assets, and sidecar test files - [Validation](./validation.md) — Schema validation, "did you mean?" suggestions, variable checks, and early regex validation diff --git a/docs/overrides.md b/docs/overrides.md index 4bc11fe..8c83dcd 100644 --- a/docs/overrides.md +++ b/docs/overrides.md @@ -51,10 +51,12 @@ Only these fields can be overridden in `environments` and `tiers`: | `fallback_models` | `string[]` | Fallback model list | | `reasoning` | `object` | `{ effort, budget_tokens }` | | `sampling` | `object` | `{ temperature, top_p, frequency_penalty, presence_penalty, stop, max_output_tokens }` | -| `response` | `object` | `{ format, stream }` | +| `response` | `object` | `{ format, stream, schema, schema_name, schema_strict }` | +| `cache` | `object` | Provider-specific cache controls (`openai`, `anthropic`, `gemini`/`google`) | | `tools` | `array` | Tool references | +| `provider_options` | `object` | Provider-specific advanced options (`anthropic`, `gemini`) | -Object fields (`reasoning`, `sampling`, `response`) are shallow-merged — individual sub-fields are replaced, but you don't need to repeat every sub-field. +Object fields (`reasoning`, `sampling`, `response`, `cache`, `provider_options`) are shallow-merged — individual sub-fields are replaced, but you don't need to repeat every sub-field. ## Applying overrides diff --git a/docs/schema.md b/docs/schema.md index 8d7d000..4743a6e 100644 --- a/docs/schema.md +++ b/docs/schema.md @@ -9,7 +9,7 @@ Prompt files use YAML front matter. This page documents every supported field. | `id` | `string` | Yes | Unique prompt identifier (e.g. `support/reply`) | | `schema_version` | `number` | Yes | Schema version — currently `1` | | `description` | `string` | No | Human-readable description of the prompt | -| `provider` | `string` | No | `openai`, `anthropic`, `gemini`, `google`, `openrouter`, `any` | +| `provider` | `string` | No | `openai`, `openai-responses`, `anthropic`, `gemini`, `google`, `openrouter`, `any` | | `model` | `string` | No | Model name (e.g. `gpt-5.4`, `claude-sonnet-4-20250514`) | | `fallback_models` | `string[]` | No | Ordered list of fallback models | | `reasoning` | `object` | No | Reasoning/thinking configuration | @@ -17,6 +17,7 @@ Prompt files use YAML front matter. This page documents every supported field. | `response` | `object` | No | Response format and streaming | | `cache` | `object` | No | Provider-specific prompt/context caching options | | `tools` | `array` | No | Tool references (strings or inline definitions) | +| `provider_options` | `object` | No | Provider-specific advanced options (`anthropic`, `gemini`) | | `mcp` | `object` | No | MCP server references | | `context` | `object` | No | Declare expected variables and history settings | | `includes` | `string[]` | No | Paths to included prompt files (relative to this file) | @@ -30,7 +31,7 @@ Prompt files use YAML front matter. This page documents every supported field. | Field | Type | Description | |-------|------|-------------| -| `provider` | `enum` | Default provider (`openai`, `anthropic`, `google`, `gemini`, `openrouter`, `any`) | +| `provider` | `enum` | Default provider (`openai`, `openai-responses`, `anthropic`, `google`, `gemini`, `openrouter`, `any`) | | `model` | `string` | Default model identifier | | `cache` | `object` | Same as prompt-level `cache` block | | `metadata` | `object` | Same as the prompt `metadata` block (`owner`, `tags`, `review_required`, `stable`) | @@ -85,12 +86,54 @@ sampling: response: format: json # text | json | markdown stream: true + schema: + type: object + properties: + answer: + type: string + schema_name: support_reply + schema_strict: true ``` | Field | Type | Description | |-------|------|-------------| | `format` | `'text' \| 'json' \| 'markdown'` | Response format | | `stream` | `boolean` | Enable streaming | +| `schema` | `object` | Portable JSON schema object for structured output | +| `schema_name` | `string` | Optional schema name (used by OpenAI/OpenAI Responses) | +| `schema_strict` | `boolean` | Strict schema enforcement toggle (OpenAI/OpenAI Responses) | + +## `provider_options` + +Provider-specific options that are intentionally non-portable: + +```yaml +provider_options: + anthropic: + top_k: 40 + tool_choice: + type: auto + gemini: + candidate_count: 1 + top_k: 32 + seed: 42 + response_schema: + type: object + response_modalities: + - TEXT + thinking_budget_tokens: 1024 +``` + +| Field | Type | Description | +|-------|------|-------------| +| `anthropic.top_k` | `number` | Anthropic `top_k` sampling control (`>= 0`) | +| `anthropic.tool_choice` | `object` | Anthropic tool choice object | +| `gemini.candidate_count` | `number` | Gemini candidate count (`> 0`) | +| `gemini.top_k` | `number` | Gemini top-k sampling control (`>= 0`) | +| `gemini.seed` | `number` | Gemini generation seed | +| `gemini.response_schema` | `object` | Gemini-native response schema | +| `gemini.response_modalities` | `string[]` | Gemini response modalities | +| `gemini.thinking_budget_tokens` | `number` | Gemini thinking budget (`> 0`) | ## `tools` @@ -223,7 +266,7 @@ tiers: model: gpt-5.4 ``` -Each environment/tier key maps to an overrides object. Overridable fields: `model`, `fallback_models`, `reasoning`, `sampling`, `response`, `cache`, `tools`. See [Overrides](./overrides.md). +Each environment/tier key maps to an overrides object. Overridable fields: `model`, `fallback_models`, `reasoning`, `sampling`, `response`, `cache`, `tools`, `provider_options`. See [Overrides](./overrides.md). ## `metadata` diff --git a/docs/vendor-schema-gap-analysis.md b/docs/vendor-schema-gap-analysis.md new file mode 100644 index 0000000..8d650ba --- /dev/null +++ b/docs/vendor-schema-gap-analysis.md @@ -0,0 +1,84 @@ +# Vendor Schema Gap Analysis (as of April 25, 2026) + +This page compares PromptOpsKit's prompt front-matter schema with currently published vendor API schema capabilities. + +Primary references: + +- OpenAI Responses API + structured outputs + prompt caching: + - https://platform.openai.com/docs/api-reference/responses/create + - https://platform.openai.com/docs/api-reference/chat/create + - https://platform.openai.com/docs/guides/structured-outputs + - https://platform.openai.com/docs/guides/prompt-caching +- Anthropic Messages API + prompt caching: + - https://platform.claude.com/docs/en/api/messages + - https://platform.claude.com/docs/en/build-with-claude/prompt-caching +- Gemini API generation + structured output + caching: + - https://ai.google.dev/api/generate-content + - https://ai.google.dev/gemini-api/docs/structured-output + - https://ai.google.dev/api/caching +- OpenRouter structured outputs + caching: + - https://openrouter.ai/docs/features/structured-outputs + - https://openrouter.ai/docs/features/prompt-caching + +## Snapshot of current PromptOpsKit schema surface + +PromptOpsKit currently models: + +- Portable prompt settings (`reasoning`, `sampling`, `response`, `tools`, `context`). +- Provider-specific options in `provider_options` (`anthropic`, `gemini`). +- Provider-specific cache controls in `cache` (`openai`, `anthropic`, `gemini` / `google`). + +See [`docs/schema.md`](./schema.md) and [`src/schema/schema.ts`](../src/schema/schema.ts). + +## Gap analysis + +### OpenAI + +| Area | Vendor capability | PromptOpsKit status | Gap | +|---|---|---|---| +| Structured outputs | `response_format: { type: "json_schema", json_schema: { name, schema, strict, description? } }` | Supported via `response.schema`, `response.schema_name`, `response.schema_strict` | **Partial**: PromptOpsKit does not expose schema-level `description`. | +| Chat vs Responses schema parity | OpenAI publishes both Chat Completions and Responses request shapes | PromptOpsKit has dedicated adapters for both (`openai` + `openai-responses`) with shared portable `response.schema*` mapping | **Partial**: API-specific fields are intentionally not fully modeled in front matter. | +| Responses conversation threading checks | Responses supports `conversation` and `previous_response_id` threading fields | PromptOpsKit exposes both via runtime `openaiResponses` options and validates they are mutually exclusive | **Partial**: validation is runtime adapter logic, not a front-matter schema construct. | +| Prompt caching | `prompt_cache_key`, `prompt_cache_retention` (`in_memory` / `24h`) | Supported via `cache.openai.prompt_cache_key`, `cache.openai.retention` | No significant gap. | +| Streaming | `stream` in request body | Supported via `response.stream` for OpenAI adapters | No significant gap. | + +### Anthropic + +| Area | Vendor capability | PromptOpsKit status | Gap | +|---|---|---|---| +| Prompt caching | Top-level automatic caching + explicit block `cache_control` with `type`/`ttl` | Supported via `cache.anthropic.mode`, `type`, `ttl`, and explicit block toggles | **Operational note**: 1h cache behavior may require vendor beta/version headers controlled by caller. | +| Tool choice / sampling extras | `tool_choice`, `top_k` | Supported via `provider_options.anthropic` | No significant gap. | +| Structured outputs | Anthropic now documents structured outputs capabilities | PromptOpsKit currently warns that `response.schema` is ignored for Anthropic | **Gap**: no first-class Anthropic structured-output mapping yet. | + +### Gemini (Google) + +| Area | Vendor capability | PromptOpsKit status | Gap | +|---|---|---|---| +| Structured outputs | `generationConfig.responseSchema` and JSON-schema alternatives | Supported via `response.schema` and `provider_options.gemini.response_schema` | **Partial**: PromptOpsKit does not expose a dedicated `response_json_schema` field for Gemini's JSON-schema-specific alternative. | +| Streaming | Endpoint-based streaming (`streamGenerateContent`) with same request schema | PromptOpsKit warns and ignores `response.stream` for Gemini adapter body | **Gap**: no endpoint-switch abstraction based on `response.stream`. | +| Caching | Managed cached resources (`cachedContents`) and request reuse via `cachedContent` | Supported reuse only via `cache.gemini.cached_content` / `cache.google.cached_content` | **Gap**: no schema surface for cache-resource lifecycle (create/list/delete) inputs; only reference by id/name. | + +### OpenRouter + +| Area | Vendor capability | PromptOpsKit status | Gap | +|---|---|---|---| +| Structured outputs | `response_format` with `json_schema` on compatible models | Supported through OpenAI-compatible adapter path (`response.schema*`) | No major schema gap for common usage. | +| Prompt caching | Provider-dependent + explicit/automatic forms (including Anthropic-style `cache_control`) | Partially supported through existing `cache` fields | **Partial**: OpenRouter-specific knobs/headers are not explicitly modeled as first-class fields. | +| Response-healing / plugins | Optional provider features outside base chat schema | Not modeled in core schema | Out of scope by design (currently). | + +## Recommended next schema additions + +If we want closer parity with currently published vendor features while preserving portability: + +1. **Add optional `response.schema_description`** for OpenAI/OpenRouter structured outputs. +2. **Optionally formalize runtime OpenAI Responses checks** in docs/types (for `conversation` vs `previous_response_id`) as a "runtime schema" contract. +3. **Add Anthropic structured-output support path** (likely in `provider_options.anthropic` first, then portable mapping). +4. **Add Gemini JSON-schema alternative field** (e.g., `provider_options.gemini.response_json_schema`). +5. **Add optional OpenRouter vendor block** under `provider_options.openrouter` for feature flags/headers that are not portable. +6. **Document runtime responsibility for vendor headers** (for capabilities gated by beta/version headers, especially Anthropic caching modes). + +## Scope and methodology + +- This analysis focuses on **published request-schema capabilities** that affect prompt front matter and adapter request shaping. +- It intentionally excludes pricing, policy, and model-availability differences except where they change request schema behavior. +- Where docs are provider-specific and evolving quickly, treat this page as a dated snapshot and re-verify against vendor docs before implementing changes.