Core logicReal OpenRouter Agent Pipeline + Per-Agent Token Optimizer, Telemetry, and Validation Hardening - #4
Conversation
There was a problem hiding this comment.
Pull request overview
This PR cuts over the backend orchestration layer from mock provider execution to real OpenRouter-backed agent calls, adding per-agent token budgeting/optimization, strict output validation, and expanded SSE + Jobs API telemetry for token/cost analysis.
Changes:
- Added OpenRouter provider implementation and wired API runtime to use it (env-configured).
- Introduced per-agent token optimizer (compression, budget guardrails, dynamic output caps) and strict Zod schema validation at agent boundaries.
- Expanded SSE
agent_completedtelemetry and added Jobs API token-usage summaries + related integration tests.
Reviewed changes
Copilot reviewed 23 out of 24 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| packages/shared/src/types/agent.types.ts | Extends AgentConfig contract to include model + token optimizer controls. |
| packages/shared/src/schemas/sse.schema.ts | Expands agent_completed payload schema with token/optimizer/provider metadata. |
| packages/agents/test/token-optimizer.test.ts | Adds unit tests for optimizer compression + budget fail-fast behavior. |
| packages/agents/test/orchestrator-failure.test.ts | Verifies schema validation failures propagate as agent failure + orchestrator rejection. |
| packages/agents/test/openrouter-provider.test.ts | Adds tests for OpenRouter JSON parsing + non-JSON error handling. |
| packages/agents/src/provider/provider.interface.ts | Extends provider interface to accept call options + expose usage/model metadata. |
| packages/agents/src/provider/openrouter.provider.ts | Implements real OpenRouter chat completion calls + JSON parsing + usage extraction. |
| packages/agents/src/provider/mock.provider.ts | Updates mock provider to match new provider interface + token fields. |
| packages/agents/src/orchestrator/orchestrator.service.ts | Emits enriched agent_completed SSE events and threads new metrics through orchestrator. |
| packages/agents/src/optimizer/token.optimizer.ts | Adds token estimation, compression passes, budget enforcement, and output token capping. |
| packages/agents/src/index.ts | Exports new provider + optimizer from @stackforge/agents. |
| packages/agents/src/config/agent.configs.ts | Defines per-agent model + budget + compression settings. |
| packages/agents/src/agents/prompts/index.ts | Adds compact per-agent prompt building with strict JSON-only instructions. |
| packages/agents/src/agents/output.schemas.ts | Defines per-agent Zod output schemas via BlueprintSchema.pick. |
| packages/agents/src/agents/base.agent.ts | Integrates optimizer + provider options + strict schema validation + telemetry fields. |
| packages/agents/package.json | Adds a Bun test script and @types/bun. |
| bun.lock | Locks @types/bun addition. |
| apps/api/test/integration.test.ts | Updates integration test expectations for failed status + token usage in job responses. |
| apps/api/src/store/job.store.ts | Adds token usage summarization + job listing helper. |
| apps/api/src/services/generate.service.ts | Switches runtime to OpenRouter provider with env validation + lazy singleton creation. |
| apps/api/src/routes/index.ts | Adds /jobs listing route. |
| apps/api/src/controllers/jobs.controller.ts | Adds list jobs controller and includes token usage in job responses. |
| apps/api/.env.example | Documents required OpenRouter env variables. |
| README.md | Documents OpenRouter runtime config, token tuning workflow, and SSE/job telemetry. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| let selectedPass = 1; | ||
|
|
||
| for (let pass = 0; pass < attempts; pass++) { | ||
| const plan = buildCompressionPlan(config.compressionLevel, pass); | ||
| const compressedInput = compressInput(input, config.maxInputTokens, plan); | ||
| const compactInput = clampJsonSize(JSON.stringify(compressedInput), config.maxInputTokens); | ||
| const tokenEstimate = estimateTokens(compactInput); | ||
|
|
||
| selectedInput = compressedInput; | ||
| selectedPrompt = compactInput; | ||
| estimatedInputTokens = tokenEstimate; | ||
| selectedPass = pass + 1; | ||
|
|
||
| if (tokenEstimate <= config.maxInputTokens) { | ||
| break; |
There was a problem hiding this comment.
compressionPasses is always at least 1 because the loop always runs and selectedPass is set to pass + 1 even when the input already fits without any trimming. That makes the telemetry misleading (a “pass” is reported even when no compression was needed). Consider tracking whether any trimming actually occurred and reporting 0 passes when the original prompt is within budget, or renaming the metric to reflect “attempts” rather than “compression passes”.
| let selectedPass = 1; | |
| for (let pass = 0; pass < attempts; pass++) { | |
| const plan = buildCompressionPlan(config.compressionLevel, pass); | |
| const compressedInput = compressInput(input, config.maxInputTokens, plan); | |
| const compactInput = clampJsonSize(JSON.stringify(compressedInput), config.maxInputTokens); | |
| const tokenEstimate = estimateTokens(compactInput); | |
| selectedInput = compressedInput; | |
| selectedPrompt = compactInput; | |
| estimatedInputTokens = tokenEstimate; | |
| selectedPass = pass + 1; | |
| if (tokenEstimate <= config.maxInputTokens) { | |
| break; | |
| let selectedPass = 0; | |
| if (estimatedInputTokens > config.maxInputTokens) { | |
| for (let pass = 0; pass < attempts; pass++) { | |
| const plan = buildCompressionPlan(config.compressionLevel, pass); | |
| const compressedInput = compressInput(input, config.maxInputTokens, plan); | |
| const compactInput = clampJsonSize(JSON.stringify(compressedInput), config.maxInputTokens); | |
| const tokenEstimate = estimateTokens(compactInput); | |
| selectedInput = compressedInput; | |
| selectedPrompt = compactInput; | |
| estimatedInputTokens = tokenEstimate; | |
| selectedPass = pass + 1; | |
| if (tokenEstimate <= config.maxInputTokens) { | |
| break; | |
| } |
| tokensUsed: response.tokensUsed, | ||
| inputTokens: response.inputTokens ?? optimized.estimatedInputTokens, | ||
| outputTokens: response.outputTokens ?? optimized.maxOutputTokens, | ||
| totalTokens: response.tokensUsed, | ||
| estimatedInputTokens: optimized.estimatedInputTokens, | ||
| compressionPasses: optimized.compressionPasses, | ||
| providerInputTokens: response.inputTokens ?? optimized.estimatedInputTokens, | ||
| providerOutputTokens: response.outputTokens ?? optimized.maxOutputTokens, | ||
| model: response.model ?? optimized.model, |
There was a problem hiding this comment.
The fallback assignments make inputTokens and providerInputTokens identical, and outputTokens and providerOutputTokens identical, so the emitted telemetry doesn’t actually distinguish “estimated” vs “provider-reported” values. Also, defaulting outputTokens to optimized.maxOutputTokens when the provider doesn’t report usage will systematically overstate output usage/cost. Consider: (1) keep inputTokens/outputTokens as optimizer estimates, (2) only set provider*Tokens when the provider reports them (otherwise 0/null), and (3) compute totalTokens consistently from the chosen sources.
| function agentCompleted( | ||
| jobId: string, | ||
| agent: AgentName, | ||
| durationMs: number, | ||
| cached: boolean, | ||
| inputTokens: number, | ||
| outputTokens: number, | ||
| totalTokens: number, | ||
| tokensUsed: number, | ||
| estimatedInputTokens: number, | ||
| compressionPasses: number, | ||
| providerInputTokens: number, | ||
| providerOutputTokens: number, | ||
| model: string, | ||
| ): SSEEvent { |
There was a problem hiding this comment.
agentCompleted(...) takes 12 positional parameters, which is easy to misorder and hard to extend safely as telemetry evolves. Consider passing a single typed payload object (or the full AgentRunResult) and constructing the SSE payload from named properties to reduce the risk of accidental field swaps.
| const original = AGENT_CONFIGS.devops.minOutputTokens; | ||
| AGENT_CONFIGS.devops.minOutputTokens = AGENT_CONFIGS.devops.tokenBudget; | ||
|
|
||
| try { | ||
| expect(() => | ||
| optimizeAgentPayload("devops", { | ||
| prompt: "x".repeat(40000), | ||
| stack: { backend: "express", database: "postgres" }, | ||
| entities: Array.from({ length: 200 }, (_, index) => ({ name: `Entity${index}` })), | ||
| }), | ||
| ).toThrow("Insufficient output token budget"); | ||
| } finally { | ||
| AGENT_CONFIGS.devops.minOutputTokens = original; | ||
| } |
There was a problem hiding this comment.
This test mutates the shared AGENT_CONFIGS singleton (devops.minOutputTokens) which can introduce cross-test flakiness if tests/files run in parallel. Prefer building a local config override (e.g., allow optimizeAgentPayload to accept an optional config object), or deep-clone/restore the entire agent config in a beforeEach/afterEach to avoid leaking state across tests.
| const response = await fetch(this.endpoint, { | ||
| method: "POST", | ||
| headers: { | ||
| Authorization: `Bearer ${this.apiKey}`, | ||
| "Content-Type": "application/json", | ||
| "HTTP-Referer": this.appUrl, | ||
| "X-Title": this.appName, | ||
| }, | ||
| body: JSON.stringify({ | ||
| model: options.model, | ||
| messages: [ | ||
| { role: "system", content: options.systemPrompt }, | ||
| { role: "user", content: options.userPrompt }, | ||
| ], | ||
| max_tokens: options.maxOutputTokens, | ||
| temperature: options.temperature, | ||
| response_format: { type: "json_object" }, | ||
| }), | ||
| }); | ||
|
|
There was a problem hiding this comment.
fetch is called without any timeout/abort handling. With real provider cutover, a hung connection can leave orchestration jobs stuck in running indefinitely (and tie up server resources). Consider adding an AbortController with a reasonable per-call timeout and surfacing a clear timeout error, so failures are deterministic and operationally safe.
| const response = await fetch(this.endpoint, { | |
| method: "POST", | |
| headers: { | |
| Authorization: `Bearer ${this.apiKey}`, | |
| "Content-Type": "application/json", | |
| "HTTP-Referer": this.appUrl, | |
| "X-Title": this.appName, | |
| }, | |
| body: JSON.stringify({ | |
| model: options.model, | |
| messages: [ | |
| { role: "system", content: options.systemPrompt }, | |
| { role: "user", content: options.userPrompt }, | |
| ], | |
| max_tokens: options.maxOutputTokens, | |
| temperature: options.temperature, | |
| response_format: { type: "json_object" }, | |
| }), | |
| }); | |
| const controller = new AbortController(); | |
| const timeoutMs = 60000; | |
| const timeoutId = setTimeout(() => controller.abort(), timeoutMs); | |
| let response: Response; | |
| try { | |
| response = await fetch(this.endpoint, { | |
| method: "POST", | |
| headers: { | |
| Authorization: `Bearer ${this.apiKey}`, | |
| "Content-Type": "application/json", | |
| "HTTP-Referer": this.appUrl, | |
| "X-Title": this.appName, | |
| }, | |
| body: JSON.stringify({ | |
| model: options.model, | |
| messages: [ | |
| { role: "system", content: options.systemPrompt }, | |
| { role: "user", content: options.userPrompt }, | |
| ], | |
| max_tokens: options.maxOutputTokens, | |
| temperature: options.temperature, | |
| response_format: { type: "json_object" }, | |
| }), | |
| signal: controller.signal, | |
| }); | |
| } catch (error) { | |
| if (error instanceof Error && (error as any).name === "AbortError") { | |
| throw new Error(`OpenRouter request timed out after ${timeoutMs}ms`); | |
| } | |
| throw error; | |
| } finally { | |
| clearTimeout(timeoutId); | |
| } |
| let selectedPrompt = JSON.stringify(input); | ||
| let estimatedInputTokens = estimateTokens(selectedPrompt); | ||
| let selectedPass = 1; | ||
|
|
||
| for (let pass = 0; pass < attempts; pass++) { | ||
| const plan = buildCompressionPlan(config.compressionLevel, pass); | ||
| const compressedInput = compressInput(input, config.maxInputTokens, plan); | ||
| const compactInput = clampJsonSize(JSON.stringify(compressedInput), config.maxInputTokens); | ||
| const tokenEstimate = estimateTokens(compactInput); | ||
|
|
||
| selectedInput = compressedInput; | ||
| selectedPrompt = compactInput; | ||
| estimatedInputTokens = tokenEstimate; | ||
| selectedPass = pass + 1; | ||
|
|
||
| if (tokenEstimate <= config.maxInputTokens) { | ||
| break; | ||
| } | ||
| } | ||
|
|
||
| if (estimatedInputTokens > config.maxInputTokens) { | ||
| throw new Error( | ||
| `Token budget exceeded for agent '${agentName}' after ${attempts} compression attempts`, | ||
| ); | ||
| } | ||
|
|
||
| const remainingBudget = config.tokenBudget - estimatedInputTokens; | ||
| if (remainingBudget < config.minOutputTokens) { | ||
| throw new Error( | ||
| `Insufficient output token budget for agent '${agentName}' after compression`, | ||
| ); | ||
| } | ||
|
|
||
| const cappedOutputTokens = Math.min(config.maxOutputTokens, remainingBudget); | ||
|
|
||
| const prompt = buildAgentPrompt(agentName, selectedInput); | ||
|
|
||
| return { | ||
| optimizedInput: selectedInput, | ||
| systemPrompt: prompt.systemPrompt, | ||
| userPrompt: prompt.userPrompt, | ||
| model: config.model, |
There was a problem hiding this comment.
estimatedInputTokens is computed from compactInput (stringified compressedInput), but the actual prompt sent to the provider is buildAgentPrompt(...).userPrompt, which may include fewer keys and therefore be significantly smaller. This can incorrectly reduce remainingBudget / cap maxOutputTokens, or even fail-fast when the real prompt would fit. Consider estimating tokens from the final userPrompt (and ideally include systemPrompt + any fixed overhead) before deriving remainingBudget and compressionPasses telemetry.
Summary
This PR upgrades the backend from mock-agent execution to real OpenRouter-powered agents and introduces a full token optimization and observability workflow.
Key Additions
What Changed
1) Real Provider Integration
2) Token Optimizer (Core Logic)
3) Prompt Strategy
4) Quality Guardrails
5) Telemetry & Tuning Loop
agent_completedtelemetry with:inputTokensoutputTokenstotalTokens6) Test Coverage (Phase 6)
@stackforge/agentstest scriptValidation Performed
All checks passing.
Breaking / Behavioral Changes
Follow-ups
agent_completed.payload.totalTokensin API integration tests