Skip to content

InferenceFlow v0.3.0 — Scheduling and GPU Readiness

Pre-release
Pre-release

Choose a tag to compare

View all tags
@manishklach manishklach released this 23 Oct 19:06
v0.3.0
60a7c8a

InferenceFlow v0.3.0 — Scheduling, engineering blog and GPU experiment readiness

Prerelease — October 23, 2026. This release packages the asynchronous execution platform's opt-in scheduling foundation, engineering website/blog and preparation for a manually approved real GPU experiment. It is a development snapshot, not a production launch, GPU benchmark, verified cost advantage or guaranteed 60-minute SLA.

Highlights

  • Deadline-aware admission and scheduling, independent adaptive concurrency, and recommendation-only capacity planning.
  • Read-only RunPod, Vast.ai and Lambda discovery contracts plus simulation-only workload-cost comparison.
  • Worker HTTP-overlap, idle, token, engine-sequence, KV-cache and GPU-memory/utilization instrumentation.
  • Reproducible sequential, fixed-concurrency and adaptive 10/100/1,000-request experiment tools, billing audits and cleanup preparation.
  • First engineering article, editable diagrams, live website validation and desktop/mobile screenshot evidence.
  • 113 automated tests passed; 1 real-GPU test intentionally skipped.

Scheduling and execution

DeadlineEstimator models compatible backlog, input/output work, warm capacity and uncertainty, retaining explicit calibration provenance. V2 admission persists best-effort predictions and can reject strict requests before reserving credit when confidence/capacity is insufficient. DispatchPolicy adds bounded deadline urgency and aging to tenant fairness, exact model affinity and concurrency limits.

AdaptiveDispatch independently adjusts available request slots using fresh telemetry, cooldown and additive increase/multiplicative decrease. Missing/stale telemetry uses conservative initial slots; critical KV pressure pauses dispatch. CapacityPlanner accounts for startup/load assumptions and configured GPU/session/spend ceilings but only recommends actions; it does not rent capacity.

The worker sends independent concurrent HTTP requests to vLLM. CPU-only HTTP tests establish four overlapping calls through the worker loop; they do not establish real GPU continuous batching. vLLM remains responsible for token-level batching, model execution and KV management. No duplicate application microbatch queue was introduced.

Providers, safety and accounting

RunPod remains the only execution-selectable provider. Optional Vast.ai/Lambda adapters normalize inventory, hardware where reported, prices and lifecycle states. Their create/delete operations are deliberately blocked, with deterministic mocked contracts and sanitized bounded read failures. Complete quote filtering excludes unknown storage/ancillary fees; offline workload comparison additionally considers compatibility, throughput assumptions, startup, availability, reliability and deadlines. It never activates routing or failover.

A separate REVIEWED_GPU_EXPERIMENT_ENABLED gate allows a reviewed private real-inference experiment without enabling PUBLIC_BETA_ENABLED. Both default false. The proposed staging overlay retains disabled public beta, Stripe and automatic provisioning. The provider watchdog now audits untracked same-prefix pods instead of deleting them: tracked identity or persistent create intent is required for cleanup ownership.

Customer wallet, pricing and settlement algorithms are unchanged. Tests exercise interruption, stale epoch/generation rejection, reservation conservation, unique usage/capture sources, tenant isolation and reconciliation. Exactly-once accepted accounting does not mean exactly-once physical GPU execution.

Instrumentation and experiment tooling

Optional metadata-only traces record HTTP overlap without prompts, outputs or credentials. Fresh worker telemetry exposes engine running/waiting sequences, token counters, KV usage, hardware utilization/memory, HTTP idle/overlap and throughput proxies. Missing measurements remain missing. Raw loopback engine metrics and NVIDIA samples are retained privately for a future experiment; process restarts and counter resets require separate measurement windows.

The inert RunPod proposal pins Qwen/Qwen3-4B model/tokenizer revision1cfa9a7208912126459214e8b04321603b3df60c and vLLM0.8.5 image digest6cf9808ca8810fc6c3fd0451c2e7784fb224590d81f7db338e7eaf3c02a33d33. It proposes one Community RTX4090, 80GB ephemeral disk, context10240, four engine/worker slots and no public model-serving ports. Artifact existence and architecture compatibility were checked; actual GPU fit/startup remain unverified.

New tools prepare deterministic synthetic corpora, validate selected policy/configuration, preserve idempotency intents, collect observations, audit per-request timing/billing and compare matched runs. Offline economics separates estimates from operator-attested invoices and includes full paid uptime/storage/ancillary costs. An independently gated exact-pod guard prepares termination at budget/time thresholds, with fresh deletion confirmation. It has only been tested with mocks and is not a provider-enforced spending cap.

Website and engineering content

The existing private ChatGPT Site includes the full article “What If AI Inference Didn't Have to Be Instant?”, blog index/navigation, author/date/reading time, accessible editable SVG architecture diagrams, canonical/social metadata and article structured data. Live homepage/blog/article navigation, mobile layout, diagrams and assets were inspected; all15 static assets returned HTTP200. Browser evidence and screenshots are included under docs/validation/2026-10-23.

Follow-up prototype-copy, contrast, homepage canonical and private-indexing fixes are included in source but remain unpublished pending separate Sites confirmation. GitHub pushes and this release do not deploy Sites or the backend. The existing Site audience remains owner-private. Unrelated unfinished console drafts are excluded.

Validation

  • Final PostgreSQL-backed suite: 113 passed, 1 real-GPU test skipped,86.73 seconds; one Starlette/AnyIO deprecation warning.
  • Ruff clean across apps/packages/scripts/tests; mypy clean across34 package/application source files.
  • Python compilation, OpenAPI export and Bash launcher syntax checks passed.
  • Website content/link/metadata checks and desktop/mobile replay/dialog/language-tab regression checks passed.
  • Mocked provider discovery, guard ownership/deletion,1,000-request drivers, interruption and economics tests require no paid resources.

The database suite uses a dedicated synthetic test database. No real GPU requests, production customer data, live payment transactions or paid resource operations were performed. Detailed verification and resolved environment failures are in docs/validation/2026-10-23/quality-report.md.

Setup, rollout and rollback

Follow README.md for the existing Docker/local stack. Install locked dependencies and GPU/tokenizer extras only where needed; apply alembic upgrade head through migration0006 and restart API/scheduler/worker together after backup/review. No additional financial migration is introduced here.

SCHEDULER_V2_ENABLED and ADAPTIVE_DISPATCH_ENABLED default false and must be configured independently. To roll back scheduling behavior, disable BOTH and restart API/scheduler; retain accepted job targets, immutable rate snapshots, observation schema and financial history. Keep ESTIMATOR_CALIBRATED=false until real evidence supports calibration. Keep PROVISIONING_ENABLED=false, STRIPE_ENABLED=false and PUBLIC_BETA_ENABLED=false. A private experiment requires its own explicit reviewed gate and later expense approval.

Known blockers and next milestone

NOT READY — BLOCKERS REMAIN. At the documented October23 quote, Community RTX4090 stock was NONE. The configured control plane was localhost HTTP, not verified private staging HTTPS. Actual placement/all-in tax/storage quote, independent supervisor credentials, remote authentication, GPU startup/memory/concurrency canaries and measured recovery remain unresolved. Secure L4/L40S alternatives were quoted but no substitution is authorized.

The10USD experiment allowance is an intention, not a RunPod hard cap. No GPU provisioning, automatic capacity, public alpha, live Stripe or backend deployment is authorized by publishing this release. Real performance, actual invoice economics, sustainable20–30% savings, hour-long deadline reliability, external OIDC/Stripe/provider behavior, restore drills and production hardening remain unverified. Vast.ai/Lambda are discovery preparation, not live production execution providers.

Detailed references

This prerelease is tagged from the readiness branch containing the blog and GPU preparation commits. PR4 and PR5 remain open; publication does not merge them into main. Review/merge and Sites publication are separate operations.

Assets 2
Loading