Summary
Add first-class OpenAI-compatible Batch, Files, and Fine-tuning endpoints to the data plane. Today these are only reachable through the opaque /passthrough/:provider/* tunnel (no token/cost attribution, no unified surface), which the competitive matrix (api7/AISIX-Cloud#873 section ⑤) marks as a headline ❌ vs LiteLLM / Portkey / Kong.
Scope (LiteLLM-baseline aligned)
Routes:
POST /v1/files (multipart upload), GET /v1/files, GET /v1/files/{id}, GET /v1/files/{id}/content, DELETE /v1/files/{id}POST /v1/batches, GET /v1/batches/{id}, GET /v1/batches, POST /v1/batches/{id}/cancelPOST /v1/fine_tuning/jobs, GET /v1/fine_tuning/jobs, GET /v1/fine_tuning/jobs/{id}, POST /v1/fine_tuning/jobs/{id}/cancel
Provider routing (the model is not in the request body for batch create):
- File upload accepts an explicit
model (form field / query / header); the returned file id is re-encoded to embed the routing model (LiteLLM's file-id encoding scheme, decode_model_from_file_id / encode_file_id_with_model in litellm/proxy/openai_files_endpoints/common_utils.py), so subsequent POST /v1/batches with that input_file_id routes automatically.
- Explicit
model / provider override on each endpoint as fallback.
- Provider coverage v1: openai family (incl. any OpenAI-compatible
api_base) + Azure OpenAI — matching LiteLLM's OPENAI_COMPATIBLE_BATCH_AND_FILES_PROVIDERS core. Vertex (GCS-based) / Bedrock (S3-based) / Anthropic native /v1/messages/batches are different wire+storage flows → separate follow-ups.
Usage/cost attribution (LiteLLM parity, per user decision — no deferral):
- On batch retrieve reaching
status=completed, download the output file JSONL, aggregate per-line usage, emit UsageEvent(s) with real token counts (once per batch id, dedup-guarded).
Auth/governance: reuse the existing caller-key auth envelope, model ACL, quota, and usage-event pipeline (same treatment as /passthrough).
Tests
- DP standalone e2e (vitest + mock upstream): upload→batch→retrieve→content round-trip, provider routing via encoded file id, cancel, fine-tuning job lifecycle, usage emission on completion.
Part of the section-⑤ gap closure for api7/AISIX-Cloud#873.
Summary
Add first-class OpenAI-compatible Batch, Files, and Fine-tuning endpoints to the data plane. Today these are only reachable through the opaque
/passthrough/:provider/*tunnel (no token/cost attribution, no unified surface), which the competitive matrix (api7/AISIX-Cloud#873 section ⑤) marks as a headline ❌ vs LiteLLM / Portkey / Kong.Scope (LiteLLM-baseline aligned)
Routes:
POST /v1/files(multipart upload),GET /v1/files,GET /v1/files/{id},GET /v1/files/{id}/content,DELETE /v1/files/{id}POST /v1/batches,GET /v1/batches/{id},GET /v1/batches,POST /v1/batches/{id}/cancelPOST /v1/fine_tuning/jobs,GET /v1/fine_tuning/jobs,GET /v1/fine_tuning/jobs/{id},POST /v1/fine_tuning/jobs/{id}/cancelProvider routing (the model is not in the request body for batch create):
model(form field / query / header); the returned file id is re-encoded to embed the routing model (LiteLLM's file-id encoding scheme,decode_model_from_file_id/encode_file_id_with_modelinlitellm/proxy/openai_files_endpoints/common_utils.py), so subsequentPOST /v1/batcheswith thatinput_file_idroutes automatically.model/ provider override on each endpoint as fallback.api_base) + Azure OpenAI — matching LiteLLM'sOPENAI_COMPATIBLE_BATCH_AND_FILES_PROVIDERScore. Vertex (GCS-based) / Bedrock (S3-based) / Anthropic native/v1/messages/batchesare different wire+storage flows → separate follow-ups.Usage/cost attribution (LiteLLM parity, per user decision — no deferral):
status=completed, download the output file JSONL, aggregate per-lineusage, emit UsageEvent(s) with real token counts (once per batch id, dedup-guarded).Auth/governance: reuse the existing caller-key auth envelope, model ACL, quota, and usage-event pipeline (same treatment as
/passthrough).Tests
Part of the section-⑤ gap closure for api7/AISIX-Cloud#873.