InferenceFlow v0.1.0 — Executable MVP

InferenceFlow v0.1.0 — Executable MVP

Release date: October 22, 2026
Channel: MVP prerelease
Default execution: deterministic mock; GPU provisioning and Stripe disabled.

Overview

This initial release implements the complete local prepaid inference flow: authenticate, submit a chat-generation batch, reserve credits, queue/schedule work, execute through a separate worker, fetch ordered outputs, meter usage, and capture/release funds atomically. It includes a real vLLM integration path, with rented-GPU validation explicitly pending.

Features

API and customer experience

• FastAPI REST endpoints with OpenAPI, request IDs, validation limits, standard errors and tenant-scoped access.
• Batches of 1–100 requests, canonical idempotency, job history/status, ordered partial results, pagination, authenticated JSONL and cancellation.
• Next.js/TypeScript dashboard for sign-in, keys, credits/reservations, sample submission, polling, downloads, tokens, charges and ledger.
• Python SDK with automatic idempotency keys, transient retries, polling, results, cancellation, catalog and wallet.

Identity and security

• Managed OIDC authorization code + PKCE, signed state, nonce and JWKS/issuer/audience/expiry validation.
• Development-only email sign-in and credit-seeding CLI.
• Users/owner memberships, one-time high-entropy API keys, keyed hashes, revocation, separate worker credentials, secure cookies and CSRF checks.
• Encrypted database content, request/token/queue/concurrency caps, content-free logs and seven-day purge.
• Registered HTTPS callbacks with public-IP validation, pinned connection IP/TLS SNI, rebinding protection and redirect rejection.

Scheduler and workers

• PostgreSQL durable queue, transactional SKIP LOCKED dispatch, separate scheduler and outbound async worker.
• Warm-model affinity, weighted tenant fairness/aging, deadline expiry, concurrency caps and KV-cache pressure admission.
• Lease epochs/fencing, heartbeat renewal, bounded retries/backoff and stale-completion rejection.
• Deterministic mock engine and private vLLM HTTP adapter with health/canary checks, explicit template/decoding behavior and revision agreement.
• Exact Qwen3-4B model/tokenizer source SHA and vLLM image digest, clearly marked as source pins rather than GPU validation.
• Reported engine queue/running counts, KV-cache pressure and NVIDIA utilization where available.

Accounting and payments

• Integer micro-USD, atomic worst-case reservations, immutable rates/ledger, unique usage, capture once and unused-credit release.
• Usage-mismatch quarantine and audited operator reconciliation or full release.
• Stripe test Checkout and raw-body signature verification, paid-session/PaymentIntent dedupe, cumulative refunds, dispute freeze and shortfall audit.
• Signed terminal-job callbacks with durable exponential retries; callback failures cannot change job money.

Deployment and operations

• Modular fake, manual-attach and Runpod REST v2 adapters.
• Default provisioning kill switch, one-GPU cap including creation intents, conservative quoted budget/session caps and independent orphan/idle/session watchdog.
• Compose PostgreSQL/migrations/API/scheduler/worker/dashboard stack and optional Prometheus.
• Metrics/alerts, optional OTLP API traces, health/readiness, CI, environment template, smoke scripts and runbooks.

Validation

Check	Recorded result
Native PostgreSQL suite	32 passed, 1 skipped
Python 3.12 Docker suite	32 passed, 1 skipped
Skipped case	Explicitly opt-in real-GPU end-to-end test
Migration downgrade/rebuild	Passed in isolated test database
Ruff / mypy	Passed
Dashboard production build / TypeScript	Passed
npm production audit	Zero reported vulnerabilities at validation time
Native three-process HTTP smoke	100 completed, zero ledger drift
Docker HTTP smoke	100 completed, zero ledger drift
Browser	Local sign-in, sample submission, succeeded state and charge observed
Exact real tokenizer	Loaded at pinned SHA; checked sample counts 18 tokens

Tests cover authentication/revocation, tenant isolation, idempotency conflicts, ordering/downloads, concurrent reservations/submissions/finalizers, rounding and ledger reconciliation, worker crash/stale-epoch recovery, retries, cancellation, partial success, expiration/late completion, signed payments, OIDC, SSRF/rebinding/redirects, callbacks, logging, body limits, retention, migrations and pressure admission.

Recorded 100-request mock workload

{
  "completed": 100,
  "captured_micro_usd": 2500,
  "usage": {"input_tokens": 3690, "output_tokens": 6900},
  "ledger_drift": 0
}

This is $0.002500 at synthetic test rates. It is not a public price, GPU benchmark, provider invoice, or commercial margin measurement.

Install and run

git clone https://github.com/manishklach/inferenceflow.git
cd inferenceflow
cp .env.example .env
docker compose up --build -d
docker compose exec api python -m packages.common.cli seed
docker compose exec api python scripts/docker_smoke.py

PowerShell can copy the template with Copy-Item. Docker Engine must be running. Dashboard: http://localhost:3000. API docs: http://localhost:8080/docs. Sign in as developer@example.local after seeding; save the generated key when displayed.

Compatibility and migrations

• Python 3.12 deployment target, with local Python 3.11 compatibility; PostgreSQL 16; Node.js 22; Compose v2.
• Schema revisions: 0001 foundation, 0002 model/rate registry, 0003 memberships, 0004 worker telemetry.
• Compose applies Alembic migrations before API startup. Native installations run the upgrade command explicitly.
• Initial release: no prior-version upgrade contract. Back up funded databases; test downgrades are not production rollback instructions.
• GitHub supplies source archives. No hosted service, PyPI package, public container registry image, or signed binary publication is claimed.

Limitations and launch gates

• No rented GPU was provisioned. Real inference, GPU memory fit/throughput, eviction recovery and GPU telemetry need approved infrastructure validation.
• Real OIDC, Stripe test Checkout, Runpod account calls and public HTTPS callbacks require external credentials/configuration. Tests use mocked external HTTP and real signature checks where applicable.
• One enabled model. Optional second model, advanced throughput prediction, organization UI, distributed rate limits and invoice ingestion are outside this implementation.
• Reservations/leases/idempotency are consolidated into durable rows; encrypted PostgreSQL columns replace object storage.
• Provider cost controls use conservative quote/uptime estimates. Independent watchdogs and provider-side account limits/alerts remain necessary during outages.
• Public rates require representative measurements, cost reconciliation and margin review.
• Public-beta security, backup/restore, model-license, prepaid-credit, refund, legal and tax review remain pending.

Safety defaults

Mock execution, provisioning disabled, Stripe disabled, beta gate disabled, one-GPU cap. Replace development secrets before deployment. Never expose vLLM publicly; use a private server and outbound worker.

No real payment credentials or billable cloud resources were used in development.

Documentation

README · Implementation status · Architecture · API · Billing · GPU deployment · Security · Operations