tazaar - Token Bazaar

Tazaar is a deployment and operations blueprint for selling private GPU capacity as an OpenAI-compatible model API.

The initial architecture is based on the earlier design discussion:

Customers
  -> CDN / WAF / HTTPS gateway
  -> New API for users, API keys, quota, pricing, billing, routing
  -> GPUStack for private GPU fleet, model deployment, inference endpoints
  -> vLLM / SGLang / llama.cpp / TensorRT-LLM workers
  -> GPU nodes

The repo starts with runnable infrastructure templates, operating docs, and a small cost-reporting tool. It is intentionally not a fork of GPUStack or New API. Those projects stay as upstream components; this repo owns the deployment, measurement, pricing, and commercial operating layer around them.

MVP Scope

The first usable version should answer one business question every day:

For each customer and model, are we making money after GPU cost, idle time, failures, and long-context usage?

MVP components:

• GPUStack as the private model and GPU orchestration layer.
• New API as the public API gateway and commercial control plane.
• SQLite and Redis for the local New API control plane.
• ClickHouse for request and usage events.
• Prometheus and Blackbox Exporter for local service health checks.
• Cost reports that compare token revenue against measured GPU cost.

Repository Layout

configs/
  pricing.example.json        Example model pricing and measured cost assumptions.
docs/
  architecture.md             System boundaries and request flow.
  metrics-and-billing.md      Required metrics, log fields, and margin math.
  mvp-roadmap.md              Build order from lab setup to limited customer beta.
examples/
  usage-events.csv            Sample usage export for cost reporting.
infra/
  k8s/minikube/               Local Kubernetes deployment for Minikube.
  terraform/digitalocean-gpustack/
                                DigitalOcean GPUStack server + one GPU worker.
scripts/
  cost_report.py              Token revenue and margin report from usage events.

Quick Start

Generate a sample margin report:

python3 scripts/cost_report.py \
  --pricing configs/pricing.example.json \
  --usage examples/usage-events.csv

Start the Minikube deployment:

make up

This starts Minikube, applies the Kubernetes manifests, and waits for the deployments.

Start local port forwarding:

make port-forward

This target stays in the foreground. Press Ctrl-C to stop forwarding.

To do both in one command:

make dev

Access the services:

New API:    http://localhost:13000
Prometheus: http://localhost:19090

Manage the local environment:

make status
make port-forward
make stop-port-forward
make down

Prometheus probes New API availability through Blackbox Exporter because this New API image returns the web app from /metrics, not Prometheus exposition text. Use this command to check the probe:

make probe

This Minikube stack is a starting point, not a production hardening checklist. Keep GPUStack on a private network and expose only New API through a real edge gateway.

Make Targets

Common targets:

make start                # Start Minikube
make stop                 # Stop port forwarding and Minikube
make deploy               # Apply Kubernetes manifests
make wait                 # Wait for all deployments
make up                   # Start, deploy, and wait
make dev                  # Start, deploy, wait, then port-forward
make status               # Show Minikube and workload status
make port-forward         # Forward New API and Prometheus in the foreground
make port-forward-bg      # Forward in the background for interactive shells
make stop-port-forward    # Stop background forwarding
make logs-new-api         # Tail New API logs
make logs-prometheus      # Tail Prometheus logs
make probe                # Query New API probe_success from Prometheus
make down                 # Delete the tazaar namespace
make delete               # Delete tazaar namespace and stop Minikube

Implementation Priorities

1. Deploy one GPUStack model and verify direct /v1/chat/completions usage.
2. Register GPUStack as an OpenAI-compatible channel in New API.
3. Validate streaming and non-streaming token usage fields match upstream logs.
4. Export request usage into ClickHouse or CSV and run daily margin reports.
5. Add Prometheus/Grafana dashboards for QPS, TTFT, TPOT, tokens/s, errors, GPU utilization, and VRAM.
6. Add customer-level rate limits, spend alerts, model access groups, and abuse controls.
7. Move from trusted beta customers to broader access only after compliance and payment workflows are handled.

Safety Boundaries

• Do not expose GPUStack or GPU worker nodes directly to the public internet.
• Do not sell anonymous access without rate limits and spend controls.
• Do not price models only by total tokens; separate input, output, context, priority, and dedicated capacity.
• Do not assume upstream usage fields are accurate until tested for both streaming and non-streaming calls.
• Treat public API resale and generated AI services as compliance-sensitive.

Minikube

Deploy the lightweight SQLite stack to Minikube:

make up
make port-forward

See docs/minikube.md for rollout and access commands.

DigitalOcean GPUStack Lab

Deploy GPUStack on DigitalOcean with one CPU server Droplet and one GPU worker Droplet:

cd infra/terraform/digitalocean-gpustack
cp terraform.tfvars.example terraform.tfvars
terraform init
terraform plan
terraform apply

See infra/terraform/digitalocean-gpustack/README.md for required variables, GPU size selection, and teardown notes.