InferenceFlow v0.3.0 — Scheduling and GPU Readiness

InferenceFlow v0.3.0 — Scheduling, engineering blog and GPU experiment readiness

Prerelease — October 23, 2026. This release packages the asynchronous execution platform's opt-in scheduling foundation, engineering website/blog and preparation for a manually approved real GPU experiment. It is a development snapshot, not a production launch, GPU benchmark, verified cost advantage or guaranteed 60-minute SLA.

Highlights

• Deadline-aware admission and scheduling, independent adaptive concurrency, and recommendation-only capacity planning.
• Read-only RunPod, Vast.ai and Lambda discovery contracts plus simulation-only workload-cost comparison.
• Worker HTTP-overlap, idle, token, engine-sequence, KV-cache and GPU-memory/utilization instrumentation.
• Reproducible sequential, fixed-concurrency and adaptive 10/100/1,000-request experiment tools, billing audits and cleanup preparation.
• First engineering article, editable diagrams, live website validation and desktop/mobile screenshot evidence.
• 113 automated tests passed; 1 real-GPU test intentionally skipped.

Scheduling and execution

DeadlineEstimator models compatible backlog, input/output work, warm capacity and uncertainty, retaining explicit calibration provenance. V2 admission persists best-effort predictions and can reject strict requests before reserving credit when confidence/capacity is insufficient. DispatchPolicy adds bounded deadline urgency and aging to tenant fairness, exact model affinity and concurrency limits.

AdaptiveDispatch independently adjusts available request slots using fresh telemetry, cooldown and additive increase/multiplicative decrease. Missing/stale telemetry uses conservative initial slots; critical KV pressure pauses dispatch. CapacityPlanner accounts for startup/load assumptions and configured GPU/session/spend ceilings but only recommends actions; it does not rent capacity.

The worker sends independent concurrent HTTP requests to vLLM. CPU-only HTTP tests establish four overlapping calls through the worker loop; they do not establish real GPU continuous batching. vLLM remains responsible for token-level batching, model execution and KV management. No duplicate application microbatch queue was introduced.

Providers, safety and accounting

RunPod remains the only execution-selectable provider. Optional Vast.ai/Lambda adapters normalize inventory, hardware where reported, prices and lifecycle states. Their create/delete operations are deliberately blocked, with deterministic mocked contracts and sanitized bounded read failures. Complete quote filtering excludes unknown storage/ancillary fees; offline workload comparison additionally considers compatibility, throughput assumptions, startup, availability, reliability and deadlines. It never activates routing or failover.

A separate REVIEWED_GPU_EXPERIMENT_ENABLED gate allows a reviewed private real-inference experiment without enabling PUBLIC_BETA_ENABLED. Both default false. The proposed staging overlay retains disabled public beta, Stripe and automatic provisioning. The provider watchdog now audits untracked same-prefix pods instead of deleting them: tracked identity or persistent create intent is required for cleanup ownership.

Customer wallet, pricing and settlement algorithms are unchanged. Tests exercise interruption, stale epoch/generation rejection, reservation conservation, unique usage/capture sources, tenant isolation and reconciliation. Exactly-once accepted accounting does not mean exactly-once physical GPU execution.

Instrumentation and experiment tooling

Optional metadata-only traces record HTTP overlap without prompts, outputs or credentials. Fresh worker telemetry exposes engine running/waiting sequences, token counters, KV usage, hardware utilization/memory, HTTP idle/overlap and throughput proxies. Missing measurements remain missing. Raw loopback engine metrics and NVIDIA samples are retained privately for a future experiment; process restarts and counter resets require separate measurement windows.

The inert RunPod proposal pins Qwen/Qwen3-4B model/tokenizer revision1cfa9a7208912126459214e8b04321603b3df60c and vLLM0.8.5 image digest6cf9808ca8810fc6c3fd0451c2e7784fb224590d81f7db338e7eaf3c02a33d33. It proposes one Community RTX4090, 80GB ephemeral disk, context10240, four engine/worker slots and no public model-serving ports. Artifact existence and architecture compatibility were checked; actual GPU fit/startup remain unverified.

New tools prepare deterministic synthetic corpora, validate selected policy/configuration, preserve idempotency intents, collect observations, audit per-request timing/billing and compare matched runs. Offline economics separates estimates from operator-attested invoices and includes full paid uptime/storage/ancillary costs. An independently gated exact-pod guard prepares termination at budget/time thresholds, with fresh deletion confirmation. It has only been tested with mocks and is not a provider-enforced spending cap.

Website and engineering content

The existing private ChatGPT Site includes the full article “What If AI Inference Didn't Have to Be Instant?”, blog index/navigation, author/date/reading time, accessible editable SVG architecture diagrams, canonical/social metadata and article structured data. Live homepage/blog/article navigation, mobile layout, diagrams and assets were inspected; all15 static assets returned HTTP200. Browser evidence and screenshots are included under docs/validation/2026-10-23.

Follow-up prototype-copy, contrast, homepage canonical and private-indexing fixes are included in source but remain unpublished pending separate Sites confirmation. GitHub pushes and this release do not deploy Sites or the backend. The existing Site audience remains owner-private. Unrelated unfinished console drafts are excluded.

Validation

• Final PostgreSQL-backed suite: 113 passed, 1 real-GPU test skipped,86.73 seconds; one Starlette/AnyIO deprecation warning.
• Ruff clean across apps/packages/scripts/tests; mypy clean across34 package/application source files.
• Python compilation, OpenAPI export and Bash launcher syntax checks passed.
• Website content/link/metadata checks and desktop/mobile replay/dialog/language-tab regression checks passed.
• Mocked provider discovery, guard ownership/deletion,1,000-request drivers, interruption and economics tests require no paid resources.

The database suite uses a dedicated synthetic test database. No real GPU requests, production customer data, live payment transactions or paid resource operations were performed. Detailed verification and resolved environment failures are in docs/validation/2026-10-23/quality-report.md.

Setup, rollout and rollback

Follow README.md for the existing Docker/local stack. Install locked dependencies and GPU/tokenizer extras only where needed; apply alembic upgrade head through migration0006 and restart API/scheduler/worker together after backup/review. No additional financial migration is introduced here.

SCHEDULER_V2_ENABLED and ADAPTIVE_DISPATCH_ENABLED default false and must be configured independently. To roll back scheduling behavior, disable BOTH and restart API/scheduler; retain accepted job targets, immutable rate snapshots, observation schema and financial history. Keep ESTIMATOR_CALIBRATED=false until real evidence supports calibration. Keep PROVISIONING_ENABLED=false, STRIPE_ENABLED=false and PUBLIC_BETA_ENABLED=false. A private experiment requires its own explicit reviewed gate and later expense approval.

Known blockers and next milestone

NOT READY — BLOCKERS REMAIN. At the documented October23 quote, Community RTX4090 stock was NONE. The configured control plane was localhost HTTP, not verified private staging HTTPS. Actual placement/all-in tax/storage quote, independent supervisor credentials, remote authentication, GPU startup/memory/concurrency canaries and measured recovery remain unresolved. Secure L4/L40S alternatives were quoted but no substitution is authorized.

The10USD experiment allowance is an intention, not a RunPod hard cap. No GPU provisioning, automatic capacity, public alpha, live Stripe or backend deployment is authorized by publishing this release. Real performance, actual invoice economics, sustainable20–30% savings, hour-long deadline reliability, external OIDC/Stripe/provider behavior, restore drills and production hardening remain unverified. Vast.ai/Lambda are discovery preparation, not live production execution providers.

Detailed references

• Complete readiness and approval package
• Live website validation, issue table and screenshots
• Quality report and changed-file index
• Provider architecture
• Scheduler architecture
• GPU deployment
• Unit economics and sensitivity
• Future private-alpha checklist

This prerelease is tagged from the readiness branch containing the blog and GPU preparation commits. PR4 and PR5 remain open; publication does not merge them into main. Review/merge and Sites publication are separate operations.

InferenceFlow v0.2.0 — Hardened MVP and Live Showcase

InferenceFlow v0.2.0 — Hardened MVP and live showcase

MVP prerelease, October 23, 2026. This release adds security and accounting fixes, a reproducible execution recording, and the complete source of the deployed Sites showcase.

Execution and recovery

• Worker generations and active epochs prevent stale workers from completing reassigned work. Artifact identities must match the dispatch they settle.
• Provider creation and watchdog reconciliation recover interrupted provisioning operations. Provisioning remains disabled by default and subject to explicit operator approval and cost caps.
• Scheduler wallet locking uses bounded batches and skips locked tenants so one busy tenant does not block unrelated work.
• Callback delivery runs in a separate process, claims durable work, and performs network requests outside database transactions.

Billing and security

• Refunds cancel or freeze affected work and record uncovered balances in an explicit debt journal.
• Session and API-key rate limits share database-backed counters across API processes.
• Production configuration requires strong bootstrap, metrics, HMAC, and encryption secrets. Key rotation supports separate HMAC and versioned encryption keys.
• Metrics require authentication; the dashboard proxy accepts only intended API routes.
• Regression tests cover concurrency, tenant isolation, billing, worker failures, lifecycle management, and the audit fixes.

Recorded flow and hosted showcase

• Seven actual browser captures show sign-in, reservation, queuing, execution, results, and settled billing.
• The walkthrough includes SDK/JSONL evidence, integration diagrams, a screenshot player, and a reproducible isolated local runner.
• Four completed mock requests and one cancellation settled at 124 microUSD, with zero remaining reservations/debt and no reconciliation drift. These are synthetic fixture rates and byte-based mock counts, not GPU measurements.
• The responsive Sites showcase includes replay controls, screenshot navigation, Python/cURL examples, and downloadable evidence. Full source and hosting identity are under sites/inferenceflow/.
• Hosted showcase currently has owner-private access. Public GitHub visibility does not change its audience. The Site is a static product demonstration; backend services have their own deployment.

Validation

• Python regression suite: 56 passed; one real-GPU test skipped.
• Ruff and mypy passed; Next.js production build passed.
• GitHub CI runs the suite against PostgreSQL 16, lint/type checks, and the dashboard build.
• Desktop/mobile showcase checks covered both billing replay presets, screenshot dialog, code tabs/copy, navigation, and viewport overflow.
• Docker Compose configuration validated. The current Windows Docker daemon was unavailable; the historical v0.1.0 Docker run is documented separately and is not evidence for this release's full container execution.

Upgrade and run

1. Back up PostgreSQL and review .env.example, docs/security.md, and docs/ops.md before upgrading.
2. Stop API, scheduler, worker, callback, and provider processes for a coordinated upgrade. Apply migrations through 0005 using alembic upgrade head (or the API container equivalent).
3. Configure the required production secrets and key versions. Start the separate callback process along with the API, scheduler, and workers; use the supplied Compose topology and operations guide.
4. Reconcile balances and verify readiness before accepting traffic. Use mock mode and seeded funds for local development.

For a fresh local platform, follow the root README's Docker quickstart. For the static Site only, run python -m http.server 8093 --bind 127.0.0.1 --directory sites/inferenceflow/dist from the repository root.

External validation still required

Rented NVIDIA GPU/vLLM execution, provider account provisioning, Stripe test-account Checkout/webhooks, and real OIDC flows require external infrastructure or credentials and have not been validated end to end. No paid capacity was provisioned and no real payment credentials were used. AMD and further cloud providers remain extension points. This prerelease does not establish production readiness or GPU performance claims.

The release source archives include the entire platform. The additional showcase ZIP contains the static Site, safe recorded evidence, and hosting metadata; it contains no runtime environment files or credentials.

InferenceFlow v0.1.0 — Executable MVP

InferenceFlow v0.1.0 — Executable MVP

Release date: October 22, 2026
Channel: MVP prerelease
Default execution: deterministic mock; GPU provisioning and Stripe disabled.

Overview

This initial release implements the complete local prepaid inference flow: authenticate, submit a chat-generation batch, reserve credits, queue/schedule work, execute through a separate worker, fetch ordered outputs, meter usage, and capture/release funds atomically. It includes a real vLLM integration path, with rented-GPU validation explicitly pending.

Features

API and customer experience

• FastAPI REST endpoints with OpenAPI, request IDs, validation limits, standard errors and tenant-scoped access.
• Batches of 1–100 requests, canonical idempotency, job history/status, ordered partial results, pagination, authenticated JSONL and cancellation.
• Next.js/TypeScript dashboard for sign-in, keys, credits/reservations, sample submission, polling, downloads, tokens, charges and ledger.
• Python SDK with automatic idempotency keys, transient retries, polling, results, cancellation, catalog and wallet.

Identity and security

• Managed OIDC authorization code + PKCE, signed state, nonce and JWKS/issuer/audience/expiry validation.
• Development-only email sign-in and credit-seeding CLI.
• Users/owner memberships, one-time high-entropy API keys, keyed hashes, revocation, separate worker credentials, secure cookies and CSRF checks.
• Encrypted database content, request/token/queue/concurrency caps, content-free logs and seven-day purge.
• Registered HTTPS callbacks with public-IP validation, pinned connection IP/TLS SNI, rebinding protection and redirect rejection.

Scheduler and workers

• PostgreSQL durable queue, transactional SKIP LOCKED dispatch, separate scheduler and outbound async worker.
• Warm-model affinity, weighted tenant fairness/aging, deadline expiry, concurrency caps and KV-cache pressure admission.
• Lease epochs/fencing, heartbeat renewal, bounded retries/backoff and stale-completion rejection.
• Deterministic mock engine and private vLLM HTTP adapter with health/canary checks, explicit template/decoding behavior and revision agreement.
• Exact Qwen3-4B model/tokenizer source SHA and vLLM image digest, clearly marked as source pins rather than GPU validation.
• Reported engine queue/running counts, KV-cache pressure and NVIDIA utilization where available.

Accounting and payments

• Integer micro-USD, atomic worst-case reservations, immutable rates/ledger, unique usage, capture once and unused-credit release.
• Usage-mismatch quarantine and audited operator reconciliation or full release.
• Stripe test Checkout and raw-body signature verification, paid-session/PaymentIntent dedupe, cumulative refunds, dispute freeze and shortfall audit.
• Signed terminal-job callbacks with durable exponential retries; callback failures cannot change job money.

Deployment and operations

• Modular fake, manual-attach and Runpod REST v2 adapters.
• Default provisioning kill switch, one-GPU cap including creation intents, conservative quoted budget/session caps and independent orphan/idle/session watchdog.
• Compose PostgreSQL/migrations/API/scheduler/worker/dashboard stack and optional Prometheus.
• Metrics/alerts, optional OTLP API traces, health/readiness, CI, environment template, smoke scripts and runbooks.

Validation

Check	Recorded result
Native PostgreSQL suite	32 passed, 1 skipped
Python 3.12 Docker suite	32 passed, 1 skipped
Skipped case	Explicitly opt-in real-GPU end-to-end test
Migration downgrade/rebuild	Passed in isolated test database
Ruff / mypy	Passed
Dashboard production build / TypeScript	Passed
npm production audit	Zero reported vulnerabilities at validation time
Native three-process HTTP smoke	100 completed, zero ledger drift
Docker HTTP smoke	100 completed, zero ledger drift
Browser	Local sign-in, sample submission, succeeded state and charge observed
Exact real tokenizer	Loaded at pinned SHA; checked sample counts 18 tokens

Tests cover authentication/revocation, tenant isolation, idempotency conflicts, ordering/downloads, concurrent reservations/submissions/finalizers, rounding and ledger reconciliation, worker crash/stale-epoch recovery, retries, cancellation, partial success, expiration/late completion, signed payments, OIDC, SSRF/rebinding/redirects, callbacks, logging, body limits, retention, migrations and pressure admission.

Recorded 100-request mock workload

{
  "completed": 100,
  "captured_micro_usd": 2500,
  "usage": {"input_tokens": 3690, "output_tokens": 6900},
  "ledger_drift": 0
}

This is $0.002500 at synthetic test rates. It is not a public price, GPU benchmark, provider invoice, or commercial margin measurement.

Install and run

git clone https://github.com/manishklach/inferenceflow.git
cd inferenceflow
cp .env.example .env
docker compose up --build -d
docker compose exec api python -m packages.common.cli seed
docker compose exec api python scripts/docker_smoke.py

PowerShell can copy the template with Copy-Item. Docker Engine must be running. Dashboard: http://localhost:3000. API docs: http://localhost:8080/docs. Sign in as developer@example.local after seeding; save the generated key when displayed.

Compatibility and migrations

• Python 3.12 deployment target, with local Python 3.11 compatibility; PostgreSQL 16; Node.js 22; Compose v2.
• Schema revisions: 0001 foundation, 0002 model/rate registry, 0003 memberships, 0004 worker telemetry.
• Compose applies Alembic migrations before API startup. Native installations run the upgrade command explicitly.
• Initial release: no prior-version upgrade contract. Back up funded databases; test downgrades are not production rollback instructions.
• GitHub supplies source archives. No hosted service, PyPI package, public container registry image, or signed binary publication is claimed.

Limitations and launch gates

• No rented GPU was provisioned. Real inference, GPU memory fit/throughput, eviction recovery and GPU telemetry need approved infrastructure validation.
• Real OIDC, Stripe test Checkout, Runpod account calls and public HTTPS callbacks require external credentials/configuration. Tests use mocked external HTTP and real signature checks where applicable.
• One enabled model. Optional second model, advanced throughput prediction, organization UI, distributed rate limits and invoice ingestion are outside this implementation.
• Reservations/leases/idempotency are consolidated into durable rows; encrypted PostgreSQL columns replace object storage.
• Provider cost controls use conservative quote/uptime estimates. Independent watchdogs and provider-side account limits/alerts remain necessary during outages.
• Public rates require representative measurements, cost reconciliation and margin review.
• Public-beta security, backup/restore, model-license, prepaid-credit, refund, legal and tax review remain pending.

Safety defaults

Mock execution, provisioning disabled, Stripe disabled, beta gate disabled, one-GPU cap. Replace development secrets before deployment. Never expose vLLM publicly; use a private server and outbound worker.

No real payment credentials or billable cloud resources were used in development.

Documentation

README · Implementation status · Architecture · API · Billing · GPU deployment · Security · Operations