feat: add OpenAI Batch API support for worker calls#309
Merged
Conversation
Add dual-provider batch queue strategy so worker calls (distillation,
curation, consolidation) route through the correct batch API based on
the upstream model's provider — Anthropic Messages Batches or OpenAI
Batch API — achieving 50% cost savings for both providers.
- llm-adapter.ts: accept { anthropic, openai } upstreams object, branch
on model.providerID for request/response format, add
normalizeOpenAIUsage() helper
- batch-queue.ts: extract BatchProvider interface, implement Anthropic
and OpenAI providers (JSONL upload → batch create → poll → download),
group flush by (authKey, providerID)
- pipeline.ts + CLI callers: pass both upstream URLs
- Tests: OpenAI provider grouping, failure fallback, full lifecycle,
JSONL format verification, normalizeOpenAIUsage coverage
- Use authFingerprint() in groupKey instead of raw credential value to avoid accidental exposure via logs/Sentry breadcrumbs - submit() now returns 'auth-error' for 401/403 (permanent disable) vs null for transient failures (fallback only, no session disable) - Remove 'cancelling' from OpenAI terminal states — it's transitional and should keep polling until it reaches 'cancelled' - Fix empty system string producing '""' instead of undefined/[] - Remove dead 'reject' field from PendingRequest (never called)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #264.
BatchProviderstrategy interface inbatch-queue.ts: Anthropic provider (existing logic, extracted) and new OpenAI provider (JSONL file upload →/v1/batches→ poll → download results). Items grouped at flush time by(authKey, providerID).llm-adapter.ts: Accepts{ anthropic, openai }upstreams object, branches onmodel.providerIDfor wire format (Anthropic Messages vs OpenAI Chat Completions).normalizeOpenAIUsage()maps OpenAI usage to Anthropic shape.Test plan
normalizeOpenAIUsagecoveragebun run typecheck— 0 errors,bun test— 1385 pass / 0 fail,bun run build— clean