token.place

Secure peer-to-peer generative AI platform

Quickstart

Prerequisites

Python 3.11+ (3.11/3.12 are recommended for relay quickstart; see macOS relay setup below)
Node.js 18+ (nvm use respects the included .nvmrc)

Create and activate a virtual environment before installing Python packages (recommended on all platforms, especially macOS where pip may be missing but pip3 is available):

git clone https://github.com/futuroptimist/token.place.git
cd token.place

# Use a known-good interpreter for relay dependencies (3.11/3.12 recommended)
python3.12 -m venv .venv || python3 -m venv .venv
# Windows PowerShell alternative: py -3.12 -m venv .venv; if unavailable, use py -3 -m venv .venv
source .venv/bin/activate   # Windows: .\.venv\Scripts\Activate.ps1
python -V
# Relay quickstart expects Python 3.11+ in this venv.
# If macOS/Homebrew still activates Python 3.9 (or another too-old version), recreate with:
#   rm -rf .venv && /opt/homebrew/bin/python3 -m venv .venv

python -m pip install --upgrade pip
python -m pip install -r config/requirements_relay.txt

Relay only (fast path): after activating .venv, run python relay.py and open http://127.0.0.1:5010/api/v1/health. Or run ./scripts/setup-relay-venv.sh to create the venv and install relay dependencies in one step.

Full development setup (compute node, tests, and pre-commit):

source .venv/bin/activate
python -m pip install -r config/requirements_server.txt
python -m pip install -r requirements.txt
npm ci
playwright install --with-deps chromium
python -m pip install pre-commit
pre-commit install
pre-commit run --all-files

On macOS, use python3 / pip3 (or the venv’s python / pip after activation) if the unversioned python / pip commands are not on your PATH.

Developer workflow quick reference

Personalise local config — ./scripts/setup.sh YOURNAME YOURFORK seeds .env.local with fork-specific defaults and updates repo metadata for your clone.
Run the services locally — python relay.py and python server.py keep iteration tight when changing Python code.
Run via containers — docker compose up --build is for local relay + server iteration; Sugarkube production uses the GHCR relay image plus OCI Helm chart instead.
Format & lint — pre-commit run --all-files (or make lint) mirrors the CI bot's formatting, linting, and quick tests.
Full test sweep — ./run_all_tests.sh (or make test) calls pytest, Playwright, npm checks, and Bandit just like CI.
Deploy Kubernetes manifests — make k8s-deploy applies legacy/local manifests under k8s/ to the active cluster context; do not use it as the Sugarkube production path.

Make targets surface the same workflows in shorthand:

make lint → run the full pre-commit suite (formatters, linters, tests)
make test → execute ./run_all_tests.sh
make desktop-parity-checks → run the local shared desktop parity entry point
make docker-build → build the relay Docker image used by remote nodes
make k8s-deploy → apply the current Kubernetes manifests

Desktop parity validation is a shared Windows/macOS release requirement. Before claiming desktop compute-node or two-node round-robin production readiness, follow docs/desktop_parity_validation.md for Windows CUDA, macOS Metal, CPU fallback, packaged resource resolution, API v1 E2EE relay lifecycle, Stop/Start, staging queue-depth, and round-robin evidence. The local deterministic entry point is:

python desktop-tauri/scripts/run_desktop_parity_checks.py

Key environment variables

Environment variables can be stored in a .env file and overridden in a .env.local file, which is ignored by git.

Variable	Default	Description
API_RATE_LIMIT	60/hour	Per-IP rate limit for user-facing API requests; API v1 compute-node control-plane routes use separate budgets
API_RELAY_CONTROL_PLANE_RATE_LIMIT	route default	Shared fallback for compute-node control-plane route budgets when a route-specific override is unset
API_RELAY_CONTROL_PLANE_REGISTER_RATE_LIMIT	240/hour	Per-server-public-key budget for authenticated compute-node register/lease refresh traffic
API_RELAY_CONTROL_PLANE_UNREGISTER_RATE_LIMIT	240/hour	Per-server-public-key budget for authenticated compute-node unregister/shutdown traffic
API_RELAY_CONTROL_PLANE_POLL_RATE_LIMIT	1200/hour	Per-server-public-key budget for authenticated compute-node poll/heartbeat traffic
API_RELAY_CONTROL_PLANE_RESPONSE_RATE_LIMIT	1200/hour	Per-client-public-key budget for authenticated encrypted compute-node response submissions
API_RELAY_CONTROL_PLANE_IP_RATE_LIMIT	10000/hour	Aggregate per-IP abuse budget shared by compute-node control-plane routes
TOKENPLACE_RATE_LIMIT_STORAGE_URI	(in-memory)	Optional shared Flask-Limiter/limits backend URI (for example Redis or Memcached) used by public and control-plane budgets
API_STREAM_RATE_LIMIT	30/minute	Per-IP rate limit applied only to streaming chat completions
SERVICE_NAME	token.place	Service identifier returned by health endpoints (whitespace-only overrides
		fall back to `token.place`)
API_DAILY_QUOTA	1000/day	Per-IP daily request quota
USE_MOCK_LLM	0	Use mock LLM instead of downloading a model (`1` to enable)
TOKEN_PLACE_ENV	development	Deployment environment (`development`, `testing`, `production`)
CONTENT_MODERATION_MODE	disabled	Set to `block` to enable request filtering before inference
CONTENT_MODERATION_BLOCKLIST	(defaults)	Comma-separated phrases added to the default safety blocklist
CONTENT_MODERATION_INCLUDE_DEFAULTS	1	Set to `0` to skip the built-in phrases when filtering requests
PROD_API_HOST	127.0.0.1	IP address for production API host
API_FALLBACK_URLS	(empty)	Comma-separated Cloudflare or other relay fallbacks tried in order
TOKEN_PLACE_RELAY_CLOUDFLARE_URLS	(empty)	Optional Cloudflare relay URLs appended to the server's relay pool

Set API_FALLBACK_URLS=https://relay.cloudflare.workers.dev/api/v1 to let the bundled clients retry through a Cloudflare-hosted relay whenever the primary endpoint is unreachable.

Set TOKEN_PLACE_RELAY_CLOUDFLARE_URLS (or TOKEN_PLACE_RELAY_CLOUDFLARE_URL for a single endpoint) so server.py can fail over to a Cloudflare tunnel when the local relays are down.

Configuration precedence

token.place automatically loads environment files before initialising the Python configuration stack. The precedence is:

.env
.env.<TOKEN_PLACE_ENV> (for example .env.production)
.env.local
File referenced by TOKEN_PLACE_ENV_FILE

Values already present in os.environ win over file-based values, so deployment platforms and local shells retain ultimate control.

The development requirements live in requirements.txt.

Content moderation hooks

Set CONTENT_MODERATION_MODE=block to enable pre-inference moderation for both /api/v1/chat/completions and /api/v1/completions. Requests containing phrases from the built-in safety blocklist (or any terms supplied via CONTENT_MODERATION_BLOCKLIST) are rejected with a standardized content_policy_violation error before they reach the model. Set CONTENT_MODERATION_INCLUDE_DEFAULTS=0 if you only want to enforce your custom blocklist.

Run the relay and server in separate terminals (with .venv activated):

python relay.py
python server.py

The relay listens on port 5010 by default (http://127.0.0.1:5010). Override with --port or RELAY_PORT.

Relay-served static assets default to production frontend mode. In this mode, requests to / (and /static/index.html) are rendered with Vue's minified production CDN build (vue.min.js) to avoid Vue development-mode warnings in relay-served HTML flows.

Desktop packaging remains a separate path: desktop-tauri/frontendDist contains the packaged React/Vite UI assets, so this Vue CDN switch applies only to relay-served static HTML and does not change desktop runtime behavior.

For local debugging, opt into development Vue with:

TOKENPLACE_FRONTEND_MODE=development python relay.py

Any value other than development/dev keeps production Vue.

Or start both services with Docker Compose:

docker compose up --build

Open http://127.0.0.1:5010 (or the port you configured) or run python client.py. For a minimal client use python client_simplified.py; it clears the screen when running interactively using ANSI codes with flushed output. Metrics are exposed at /metrics.

CI pass criteria

All pull requests must:

run pre-commit run --all-files
pass npm run lint
pass npm run type-check
pass npm run build
pass npm run test:ci
pass pytest -q tests/test_security.py
pass bandit -r . -lll with no medium or high findings
keep Dependabot, CodeQL, and secret-scanning badges in this README

See docs/TESTING.md for the full testing guide.

vision

There are tons of personal computers and homelabs out there with lots of compute that remain idle. This project aims to create a marketplace of people with spare compute and people with needs for compute. Note that this is not a financial marketplace -- this is intended to be a public good. If it takes off is anyone's guess, but I'll donate whatever compute I can in the meantime once this is up and running.

For a quick orientation to the repository layout and key docs, see docs/ONBOARDING.md. For a directory-by-directory atlas, visit docs/REPO_MAP.md.

Desktop direction

The desktop-tauri/ app is the forward-looking desktop path, but it is currently an MVP and does not yet replace server.py. The canonical migration sequence is documented in docs/roadmap/desktop_compute_node_migration.md.

API v1 E2EE architecture baseline (v0.1.0)

This baseline applies to the active v0.1.0 relay/client-server runtime path (including distributed relay and desktop integration paths being finalized for v0.1.0).
API v1 is the active API for v0.1.0 and the only approved runtime integration target.
API v1 is non-streaming for relay/client-server inference paths; return responses only after full model generation is complete.
Do not add streaming to API v1 for active relay/client-server paths.
API v2 exists but is incomplete; do not route runtime traffic through API v2 until API v1 is launched and v0.1.0 is finalized.
If later sections of this README show API v2 streaming or /api/v2/chat/completions examples, treat them as experimental/reference-only and not as approved runtime integration guidance for v0.1.0.
Deprecated legacy relay endpoints: /sink, /faucet, /source, /retrieve, /next_server. Do not use, extend, or reintroduce these as active production fallbacks.
Relay-blind E2EE remains mandatory: relay surfaces may see ciphertext + safe routing metadata only, and must never store or expose plaintext model payload content.

Known migration gap: relay.py, desktop-tauri paths, and relay landing-page HTML chat flow are not fully aligned yet; some E2E pieces still hit legacy routes. This migration is tracked as follow-up implementation work.

See docs/architecture/api_v1_e2ee_relay.md.

Canonical entrypoints

server.py is the canonical compute-node server entrypoint.
relay.py is the canonical relay entrypoint.
server/server_app.py is retained only as a legacy compatibility shim that delegates to server.py.

Deployment topology

Current / legacy local flow (today): local or self-hosted relay.py + server.py on the deprecated legacy sink/source contract (historical compatibility only; not for active production paths).
Near-term multi-node legacy flow (deprecated/historical migration context only): multiple compute nodes (including desktop-tauri after parity) register through relay.py using the same legacy contract until API v1 distributed migration is complete.
Future distributed API v1 flow (post-parity): distributed compute migrates to API v1-aligned contracts after parity and operational readiness gates are complete.
Operator platform targets: Windows 11 + NVIDIA/CUDA and macOS Apple Silicon + Metal first, CPU fallback always supported, Raspberry Pi as a later low-power compute-node target.

sugarkube deployment

relay.py is the only token.place runtime component deployed in Sugarkube for the near-term phase. server.py and desktop/host compute nodes remain external.

Current relay operations model in Sugarkube:

one pod, one Gunicorn worker, one replica
in-memory relay state (registrations/queued messages/replies)
accepted state loss on pod replacement until shared-state architecture lands

The canonical token.place Helm chart now defaults to strict single-pod rollout semantics for this stateful relay phase by rendering replicaCount: 1 and strategy.type: Recreate.

Artifact references:

Relay image: ghcr.io/futuroptimist/tokenplace-relay
OCI chart: oci://ghcr.io/futuroptimist/charts/tokenplace
Chart metadata: chart package version 0.1.3 preserves appVersion: "0.1.1"; the historical v0.1.0 Git tag and ghcr.io/futuroptimist/tokenplace-relay:v0.1.0 image remain examples of immutable release artifacts, not a current version-alignment requirement.
Preferred staging/prod tag: immutable main-<shortsha> copied from the ci-image.yml workflow summary (main-latest is convenience-only)
Canonical release tag after pushing a Git release tag is the matching semver image tag (example: v0.1.0 -> ghcr.io/futuroptimist/tokenplace-relay:v0.1.0)
Pre-publish gate: ci-helm.yml checks whether the current chart version already exists at oci://ghcr.io/futuroptimist/charts/tokenplace. It publishes only new chart versions when chart source files changed, skips unchanged already-published versions as a no-op, and fails changed chart source with an already-published version so maintainers bump charts/tokenplace/Chart.yaml.

Sugarkube values, wrappers, and operator workflows are maintained in the Sugarkube repo. The current app-specific deploy command is:

just tokenplace-oci-deploy env=staging tag=main-REPLACE_SHORTSHA

Once Sugarkube P5 lands, the equivalent generic command is:

just app-deploy app=tokenplace env=staging tag=main-REPLACE_SHORTSHA

GHCR-first release workflow: docs/ops/sugarkube-release.md
v0.1.2 observability release requirement: docs/releases/v0.1.2.md
Relay onboarding: docs/relay_sugarkube_onboarding.md
Environment runbooks:

End-to-end encryption powered by RSA and AES
Cross-platform Python server and JavaScript client
Inline vision analysis for base64 image attachments via the API v2 chat endpoint
Comprehensive tests and CI via GitHub Actions
AGENTS.md lists repo helpers for LLMs
llms.txt provides machine-readable context
CLAUDE.md summarizes Claude integration tips

roadmap

Canonical migration plan: docs/roadmap/desktop_compute_node_migration.md.

streaming usage

token.place supports server-sent events (sse) for plaintext requests starting with the api v2 chat completions endpoints. when the stream flag is set to true, the /api/v2/chat/completions and /v2/chat/completions routes emit incremental chunks that match openai's event format so existing clients can subscribe without code changes.

✅ encrypted chat payloads now stream encrypted chunks when you call the /api/v2/chat/completions endpoint with encrypted=true alongside stream=true. ❌ api v1 chat endpoints remain json-only; requests that include stream=true return an error. use /api/v2/chat/completions (or /v2/chat/completions) for server-sent events.

Encrypted streaming events use the same OpenAI-style chat.completion.chunk payloads as plaintext responses, but each SSE data line contains an envelope shaped like:

{"event": "delta", "encrypted": true, "data": {"ciphertext": "...", "cipherkey": "...", "iv": "..."}}

Clients can reuse CryptoClient.decrypt_message (or equivalent) to decrypt the payload before reading the inner choices[0].delta entries, preserving API compatibility with existing OpenAI SDKs.

example curl request

curl \
  -N \
  -H "content-type: application/json" \
  -H "authorization: bearer $TOKEN_PLACE_API_KEY" \
  -d '{
        "model": "gpt-5-chat-latest",
        "stream": true,
        "messages": [
          {"role": "user", "content": "summarize the roadmap"}
        ]
      }' \
  http://localhost:5050/api/v2/chat/completions

consuming the stream in python

import requests

response = requests.post(
    "http://localhost:5050/v2/chat/completions",
    headers={"Authorization": "Bearer $TOKEN_PLACE_API_KEY"},
    json={
        "model": "gpt-5-chat-latest",
        "stream": True,
        "messages": [{"role": "user", "content": "give me a streaming demo"}],
    },
    stream=True,
    timeout=30,
)

for line in response.iter_lines():
    if not line:
        continue
    if line.startswith(b"data: "):
        payload = line.removeprefix(b"data: ")
        if payload == b"[DONE]":
            break
        print(payload.decode("utf-8"))

each chunk includes the role delta, message content, optional tool calls, and a final [DONE] marker. clients can accumulate the delta.content strings to display streaming completions in their ui.

installation

virtual environment

create a virtual environment:

$ python -m venv .venv

Depending on your environment, you may need to replace python in the above command with python3.

On Debian-based distributions, you may additionally need to install venv first, if it's not already installed:

apt install python3-venv

activate the virtual environment:

unix/linux/macos

source .venv/bin/activate

windows

PowerShell:

.\.venv\Scripts\Activate.ps1

Command Prompt:

.\.venv\Scripts\activate.bat

If PowerShell activation doesn't work (e.g. Activate.ps1 cannot be loaded because running scripts is disabled on this system), run the following in PowerShell (no administrator rights required for -Scope CurrentUser):

Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser

install dependencies

You may be missing some CMAKE dependencies

on Debian/Ubuntu you can run the following

sudo apt-get install build-essential cmake

on macOS, install the Xcode command line tools and CMake via Homebrew:

xcode-select --install  # if not already installed
brew install cmake

macOS relay-only quick start

Use this path when you only need relay.py (no local llama-cpp-python build):

Install Python 3.12 (Homebrew’s default python3 may be 3.14, which lacks wheels for some pinned packages):
```
brew install python@3.12
```

Create a venv and install relay dependencies:

cd token.place
./scripts/setup-relay-venv.sh
# or manually:
python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install -r config/requirements_relay.txt

Start the relay and verify health:

python relay.py
curl http://127.0.0.1:5010/api/v1/health

If pip install fails building Pillow, install JPEG support (brew install jpeg) or use Python 3.12 as above so pip can install a prebuilt wheel. If you used python3 -m venv, recreate the venv with python3.12 -m venv .venv to avoid unsupported Homebrew default interpreters.

On other Linux distributions, use your package manager to install the equivalents of build-essential and cmake. For example on Fedora:

sudo dnf install gcc-c++ make cmake

then, run:

pip install -r config/requirements_server.txt  # server/API dependencies
pip install -r config/requirements_relay.txt   # relay-only dependencies

For JavaScript dependencies, run:

npm ci

troubleshooting `llama_cpp_python` builds

If pip install fails while building llama_cpp_python with an error such as CMAKE_CXX_COMPILER not set or Could not find compiler set in environment variable CXX, it usually means a C++ compiler is missing from your system. Install g++ (e.g. via build-essential on Debian/Ubuntu) and ensure it is on your PATH or set CXX=g++ before running pip install. The hardware acceleration section below describes how to reinstall llama_cpp_python with GPU support once your build environment is set up correctly.

hardware acceleration

If you want to also utilize your GPU (instead of just your CPU), follow these platform-specific instructions.

windows

This is the resource I used to get things finally working: https://medium.com/@piyushbatra1999/installing-llama-cpp-python-with-nvidia-gpu-acceleration-on-windows-a-short-guide-0dfac475002d

Summarizing:

Prerequisites

Visual Studio with
- C++ CMake tools for windows
- C++ core features
- Windows 10/11 SDK
CUDA Toolkit

The next steps need to be executed in the same virtual environment you set up above. You'll see something like (.venv) on the bottom left in your terminal (may not be true on all platforms in all terminals).

This will replace the llama-cpp-python you installed via pip install -r requirements.txt and will instruct it to use cuBLAS.

if you're using Command Prompt

set CMAKE_ARGS=-DGGML_CUDA=on
set FORCE_CMAKE=1
pip install llama-cpp-python --force-reinstall --upgrade --no-cache-dir --verbose

if you're using Powershell

$env:CMAKE_ARGS = "-DGGML_CUDA=on"
$env:FORCE_CMAKE=1
pip install llama-cpp-python --force-reinstall --upgrade --no-cache-dir --verbose

Note: The compilation process can take 5-20 minutes depending on your system. The --verbose flag shows build progress, but there may still be periods with no visible output. This is normal - the compiler is working in the background. As long as your system shows CPU activity, the process is still running.

when you run server.py next, you'll see BLAS = 1 in a collection of lines that looks like this:

AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 0 | VSX = 0 |

This indicates that server.py can correctly access your GPU resources.

llama-cpp-python is initialized like this:

llm = Llama(
        model_path=model_path,
        n_gpu_layers=-1,
        n_ctx=8192,
        chat_format="llama-3"
    )

n_gpu_layers instructs llama to use as much of your GPU resources as possible.

macos

Install the Homebrew package manager if you haven't already, then reinstall llama-cpp-python with Metal support:

brew install cmake
CMAKE_ARGS="-DLLAMA_METAL=on" FORCE_CMAKE=1 pip install llama-cpp-python --force-reinstall --upgrade

unix/linux

Most distributions can compile llama-cpp-python with OpenBLAS:

sudo apt-get install build-essential cmake libopenblas-dev
CMAKE_ARGS="-DLLAMA_OPENBLAS=on" FORCE_CMAKE=1 pip install llama-cpp-python --force-reinstall --upgrade

Running the servers

Dev-like environment

You'll need a way to switch between terminal tabs (e.g. tmux, VS Code terminal tabs).

Launch the relay, which runs on http://127.0.0.1:5010 by default:

python relay.py

The relay listens on port 5010 (override with --port or RELAY_PORT). It automatically connects to the default server address baked into the project, so no environment variables are required.

In a separate terminal, launch the server, which binds to 0.0.0.0 on port 3000 (accessible via http://localhost:3000):

python server.py

Compute-node entrypoints now default their relay target to https://token.place. For local development, pass an explicit override such as --relay_url http://127.0.0.1:5010 or --relay_url http://localhost:5010.

For testing with mock LLM (faster startup):

python server.py --use_mock_llm

Configuring relay upstream server nodes

token.place is bootstrapping its first community LLM cluster with volunteer-run server.py nodes. The pioneer machine still lives on futuroptimist's gaming PC, but the relay now accepts a list of upstream hosts so new contributors can join without code changes.

When the Raspberry Pi relay cluster runs in production, set TOKEN_PLACE_RELAY_UPSTREAMS before launching relay.py to provide a comma-separated (or JSON array) list of volunteer nodes:

export TOKEN_PLACE_ENV=production
export TOKEN_PLACE_RELAY_UPSTREAMS="https://gaming-pc.local:8000,https://your-node.example.com:8443"
python relay.py

Config normalises these URLs, keeps the historical gaming PC entry as the default, and surfaces any secondary nodes from /api/v1/relay/server-nodes so relay operators can verify who is online. The legacy PERSONAL_GAMING_PC_URL variable still works; it is treated as shorthand for a single-entry upstream list.

Zero-trust relay verification

token.place now ships with an opt-in challenge/response layer for compute nodes. Set TOKEN_PLACE_RELAY_SERVER_TOKEN before launching the relay and for deprecated legacy /sink or /source compatibility flows only, include an X-Relay-Server-Token header that matches the configured value. Requests missing the header are rejected with an HTTP 401 so unknown machines can no longer impersonate trusted servers. Sensitive tokens are stripped when saving config files, so store them in environment variables instead of config.json.

The bundled RelayClient automatically reads the same configuration and sends the header, so volunteer operators only need to export the token once:

export TOKEN_PLACE_RELAY_SERVER_TOKEN="rotate-me-often"
python relay.py --host 0.0.0.0
# on the compute node
python server.py --relay_url http://relay.example.com --relay_port 5010

Clients remain zero-auth: they never see or transmit the relay token. This keeps the network open for end users while letting operators quarantine suspicious server nodes using cryptography instead of static passwords.

Once that upstream list is stable, export TOKEN_PLACE_RELAY_CLUSTER_ONLY=1 before launching server.py. The background RelayClient will refuse to talk to localhost and instead require at least one upstream derived from TOKEN_PLACE_RELAY_UPSTREAMS, relay.additional_servers, or the relay.server_pool values surfaced by config. This keeps production traffic pinned to the k3s pod rather than silently falling back to the gaming PC.

Using the Application

relay.py acts as a proxy between the client (including, but not limited to, this repo's client.py) and server.py, obfuscating each other's public IP from each other, solving one of the big limitations of P2P networks (e.g. for .torrents). The relay.py provides end-to-end encryption for communication between server and client, ensuring that your messages are private even from the relay itself.

server.py runs on volunteers' machines and hosts the LLM model. It serves inference requests forwarded by the relay while keeping the server's network details private.

You can test things out using the simple command-line client, client.py:

python client.py

Relay batching

For deprecated legacy compatibility only, operators running server.py can include a max_batch_size integer in /sink polls to retrieve multiple pending jobs at once. The relay removes up to that many queued faucet requests, returns the first item via the legacy client_public_key/chat_history fields, and exposes the full batch under a batch array for upgraded workers. Leaving max_batch_size unset preserves the single-request behavior.

Type your message when prompted and press Enter. All of this is now happening on your local hardware, thanks to llama-cpp-python, a binding for llama.cpp.

To exit, press Ctrl+C/Cmd+C.

Alternatively, you can visit http://127.0.0.1:5010 in your browser to use the web interface.

Raspberry Pi 5 deployment

For a complete walkthrough of the Raspberry Pi 5 setup—including hardware recommendations, Sugarkube deployment, k3s cluster steps, and troubleshooting (including rpi-clone prompts)—see docs/RPI_DEPLOYMENT_GUIDE.md.

If you're booting via the sugarkube Pi image, deploy token.place via Sugarkube's Helm OCI install/upgrade flow using the published token.place chart and image tags (see the Sugarkube runbooks linked above), rather than local chart bundle copies.

Testing

The project includes a comprehensive test suite to ensure functionality and prevent regressions.

Running Tests

Run all tests:

python -m pytest

To execute all available test suites (including JavaScript and end‑to‑end tests), use:

./run_all_tests.sh

or on Windows:

./run_all_tests.ps1

Run specific test files:

python -m pytest tests/test_api.py
python -m pytest tests/test_e2e_network.py

Run tests with coverage report:

TEST_COVERAGE=1 ./run_all_tests.sh
# Coverage results are uploaded to Codecov on CI

If you don't see coverage comments on your pull requests, install the Codecov GitHub App on your fork. Every pull request automatically runs this test suite in GitHub Actions, so you can rely on the status checks for pass/fail information.

Tag @claude in any pull request or issue to invoke the automated Claude PR Assistant for implementation help.

Test Categories

Unit Tests: Test individual components in isolation
- Core crypto functionality (RSA/AES encryption)
- Client message handling
- Server request processing
Integration Tests: Test component interactions
- Client-server communication
- Relay server message passing
- Multi-step conversation handling
End-to-End Tests: Test complete workflows from client to server
- Chat functionality through the relay
- Browser-based UI testing with Playwright
- API endpoints with encrypted communication
API Tests: Test the OpenAI-compatible API endpoints
- Models listing
- Chat completions
- Regular completions
- Error handling
Real LLM Tests: Test with actual LLM models
- Model file verification with checksums
- Single-turn inference
- Multi-turn conversations
- Error handling and recovery
Mock Testing: Mock mode for faster development and testing
- Models can be swapped with mock implementations
- Test complex scenarios without real inference

Real LLM Testing

To run tests with the actual LLM model:

python -m pytest tests/test_real_llm.py -v

This test will:

Automatically download the model file (~4GB) if it doesn't exist
Verify the model file with SHA-256 checksum
Cache the model locally to avoid re-downloading
Run inference with the real LLM model

For more comprehensive real LLM testing:

python -m pytest tests/test_real_llm_validation.py -v

This performs additional validation including:

Multi-turn conversation testing
Complex reasoning validation
Error handling and recovery tests

Note: The first run may take several minutes to download the model file. Subsequent runs will use the cached model file if it passes checksum verification.

If you want to disable the test to avoid downloading, edit tests/test_real_llm.py and set:

RUN_REAL_LLM_TEST = False

Project Documentation for LLMs

See AGENTS.md for a list of repo helpers. For LLMs, llms.txt contains the same helper summary in plain text. This project includes an llms.txt file in the root directory that provides structured information for Large Language Models (LLMs) about the project. Following the llms.txt specification, this file helps LLMs understand the codebase structure, available components, and key documentation links.

The llms.txt file includes:

Project overview and description
Links to setup and installation instructions
Core component descriptions
Testing documentation links
Development guidelines

When working with LLMs to understand or develop this codebase, reference the llms.txt file for comprehensive context.

Naming Conventions

Always stylize the project name as lowercase token.place (not Title case "Token.place") to emphasize that it is a URL. For complete stylization guidelines, see docs/STYLE_GUIDE.md.

API (OpenAI-compatible)

The token.place API is designed to be compatible with the OpenAI API format, making it easy to integrate with existing applications that use OpenAI's services.

API v2 extends the surface with adapter-aware metadata. Fine-tuned variants such as llama-3-8b-instruct:alignment inherit the base weights and automatically prepend their alignment charter as a system prompt. OpenAI SDKs can opt into domain-specific behaviour by selecting the derived model ID. The curated v2 catalogue, including deployment notes for RTX 4090-class hardware, is documented in docs/api_v2_model_catalog.md.

API Endpoints

All routes are served under /api/v1 (preferred) and are also available at /v1 for compatibility with standard OpenAI clients. Set the base URL to https://token.place/api/v1.

Browser CORS contract

API v1 is directly callable from browser applications on arbitrary origins. token.place owns this CORS behavior in the application so staging, production, and self-hosted deployments share the same contract: public API v1 responses under /api/v1/* and the OpenAI-compatible /v1/* aliases return Access-Control-Allow-Origin: *. Calls are non-credentialed; browser clients must not send cookies or token.place authorization headers. JSON POST clients should send Content-Type: application/json. API v2 remains outside this API v1 launch contract.

Minimal browser example:

const response = await fetch("https://token.place/api/v1/chat/completions", {
  method: "POST",
  headers: {
    "Content-Type": "application/json",
    Accept: "application/json",
  },
  body: JSON.stringify({
    model: "llama-3-8b-instruct",
    messages: [{ role: "user", content: "Hello from the browser!" }],
  }),
});

const completion = await response.json();
console.log(completion);

List Models

GET /api/v1/models
# or
GET /v1/models

Returns a list of available models.

API v1 catalogue: token.place intentionally restricts /api/v1/models to the llama-3-8b-instruct base model (backed by the Meta Llama 3.1 8B Q4_K_M weights) and its safety-tuned llama-3-8b-instruct:alignment adapter. The broader RTX 4090-ready line-up lives behind /api/v2/models.

Get Model

GET /api/v1/models/{model_id}
# or
GET /v1/models/{model_id}

Returns information about a specific model.

Chat Completions

POST /api/v1/chat/completions
# or
POST /v1/chat/completions

Creates a completion for chat messages.

Request body:

{
  "model": "llama-3-8b-instruct",
  "messages": [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Hello!"}
  ]
}

Text Completions

POST /api/v1/completions
# or
POST /v1/completions

Traditional completions API returning text completion data.

Request body:

{
  "model": "llama-3-8b-instruct",
  "prompt": "Write a poem about AI",
  "max_tokens": 256
}

Health Check

GET /api/v1/health
# or
GET /v1/health

Returns the service's readiness metadata. The JSON payload includes status, version, service, and a Unix timestamp. Override the reported service identifier by setting the SERVICE_NAME environment variable; blank or whitespace-only overrides are ignored so the default token.place label is preserved.

Image Generations

POST /api/v1/images/generations
# or
POST /v1/images/generations

Creates a deterministic PNG using the local Pillow-based renderer. The endpoint is compatible with OpenAI SDK helpers that expect a b64_json payload.

Request body:

{
  "prompt": "Neon skyline over a calm ocean",
  "size": "256x256",
  "seed": 42
}

Response body:

{
  "created": 1731976800,
  "size": "256x256",
  "data": [
    {
      "b64_json": "iVBORw0KGgoAAAANSUhEUgAA...",
      "revised_prompt": "Neon skyline over a calm ocean"
    }
  ],
  "seed": 42
}

If size is omitted the renderer defaults to 512x512. Provide an integer seed to generate reproducible art assets for offline demos or unit tests.

Community Provider Directory

GET /api/v1/community/providers

Lists community-operated relay nodes and server operators that have opted into the public registry. Each entry includes the provider identifier, advertised region, contact details, current status, and the exposed endpoint URLs so clients can preselect a compatible provider. A representative latency measurement may also be included to help clients pick a nearby relay.

Example response snippet:

{
  "object": "list",
  "data": [
    {
      "id": "local-dev",
      "name": "Local Development Node",
      "region": "local",
      "status": "active",
      "endpoints": [
        {"type": "relay", "url": "http://localhost:5010"},
        {"type": "server", "url": "http://localhost:3000"}
      ]
    }
  ],
  "metadata": {
    "updated_at": "2025-02-15T00:00:00Z"
  }
}

Directory entries with blank or whitespace-only identifiers, names, or regions are rejected when the registry loads so clients never receive placeholder records.

Community Model Leaderboard

GET /api/v1/community/leaderboard

Aggregates community ratings for deployed models, returning the highest-rated experiences first. Each entry reports the average rating, number of submitted votes, and the most recent feedback timestamp so client applications can highlight trending models.

Example response snippet:

{
  "entries": [
    {
      "model_id": "anthropic/claude-3.5-sonnet",
      "average_rating": 4.8,
      "ratings_count": 27,
      "last_feedback_at": "2024-12-19T09:10:00Z"
    }
  ],
  "updated": "2024-12-19T09:10:00Z"
}

Community Contribution Queue

POST /api/v1/community/contributions

Allows community operators to offer spare compute resources for the shared relay network. Submissions are validated server-side, assigned a UUID, and appended to a JSONL queue so maintainers can review and onboard new providers.

Request body:

{
  "operator_name": "Compute Collective",
  "region": "us-west",
  "availability": "weekends",
  "capabilities": ["openai-compatible", "gpu"],
  "contact": {"email": "ops@example.org"},
  "hardware": "2x RTX 4090",
  "notes": "Can scale to 4 nodes with notice"
}

Response body:

{
  "status": "queued",
  "submission_id": "82b900a7-1c05-4e2a-8ce0-3b18a835adcb"
}

For deployments that need to relocate the queue file, set TOKEN_PLACE_CONTRIBUTION_QUEUE to an absolute path. The server will create the file if it does not exist and append one JSON document per line.

Authorising community-operated servers

Once an operator is ready to host server.py, generate an invitation token and expose it to the relay by setting TOKEN_PLACE_RELAY_SERVER_TOKENS (comma or newline delimited) before launching relay.py. Each joined node must supply the matching token via the TOKEN_PLACE_RELAY_SERVER_TOKEN environment variable, which the legacy relay client forwards to deprecated /sink and /source endpoints as the X-Relay-Server-Token header. Requests without a valid token are rejected, preventing uninvited nodes from queueing or retrieving encrypted workloads while still keeping the workflow simple for approved operators.

Community Contribution Summary

GET /api/v1/community/contributions/summary

Summarises queued community contributions so maintainers can understand incoming capacity at a glance. Returns the total number of submissions, a sorted list of participating regions, the occurrence count for each advertised capability, and the timestamp of the most recent submission.

Example response:

{
  "object": "community.contribution_summary",
  "total_submissions": 2,
  "regions": ["eu-central", "us-west"],
  "capability_counts": {"gpu": 2, "openai-compatible": 1},
  "last_submission_at": "2025-01-15T09:30:00Z"
}

End-to-End Encryption

For enhanced privacy, you can use end-to-end encryption with the API:

Get the server's public key:

GET /api/v1/public-key
# or
GET /v1/public-key

If you need to invalidate the existing key pair (for example after suspected compromise), rotate the credentials:

POST /api/v1/public-key/rotate
# or
POST /v1/public-key/rotate

# Include your operator token via either header:
# Authorization: Bearer <token>
# X-Token-Place-Operator: <token>

Configure the server-side secret by setting `TOKEN_PLACE_OPERATOR_TOKEN` (or
`TOKEN_PLACE_KEY_ROTATION_TOKEN`) in the environment before starting the API.

Encrypt your request with the server's public key
Send your encrypted request:

{
  "model": "llama-3-8b-instruct",
  "encrypted": true,
  "client_public_key": "YOUR_PUBLIC_KEY_HERE",
  "messages": {
    "ciphertext": "ENCRYPTED_DATA_HERE",
    "cipherkey": "ENCRYPTED_AES_KEY_HERE",
    "iv": "INITIALIZATION_VECTOR_HERE"
  }
}

client_public_key may be provided as a PEM-formatted string or a base64-encoded key.

The server will encrypt its response with your public key, ensuring end-to-end encryption.

New: When encrypting high-value assets such as model weights or inference payloads, call encrypt(..., cipher_mode="GCM", associated_data=...) to switch to AES-GCM. The response payload includes an additional tag field alongside ciphertext and iv, providing authenticated encryption without breaking compatibility with existing AES-CBC clients.

System Architecture

The project follows a distributed architecture with end-to-end encryption (see docs/ARCHITECTURE.md):


┌───────────────┐    Encrypted     ┌─────────────────┐    Encrypted     ┌─────────────────┐
│               │    Requests      │                 │    Requests      │                 │
│  Web Browser  │ ───────────────► │  Relay Server   │ ───────────────► │  Server (LLM)   │ ◄─── LLM Model File (~4GB)
│  or CLI       │ ◄─────────────── │  (lightweight)  │ ◄─────────────── │                 │      (downloaded locally)
│               │    Encrypted     │                 │    Encrypted     │                 │
└───────────────┘    Responses     └─────────────────┘    Responses     └─────────────────┘
                                           │
                                           │ can scale to
                                           ▼
                                   ┌─────────────────┐
                                   │  Multiple       │
                                   │  Relay Servers  │
                                   │  & LLM Servers  │
                                   └─────────────────┘

The architecture consists of:

Clients (Web browsers, CLI tools) - Send encrypted requests to relay servers
Relay Servers - Forward encrypted requests/responses, hiding IP addresses of clients and servers
LLM Servers - Run inference on local hardware (GPU/CPU), download and use LLM models, and return encrypted results
LLM Models - Downloaded and cached locally on each LLM server

All communication uses hybrid RSA/AES encryption to ensure end-to-end security, preventing relay servers from accessing the content of messages. The relay server only knows which client and server are communicating, but not what they're saying.

The API is compatible with the OpenAI format for easy integration with existing tools and libraries.

Cross-Platform Support

token.place now has full cross-platform support for Windows, macOS, and Linux. This includes:

Platform-specific path handling for user data, configs, and logs
Native launcher scripts for each platform
Docker containerization for consistent deployment
Cross-platform testing framework
Automated desktop installers for Windows (.exe) and macOS (.dmg) built via GitHub Actions

For detailed information on cross-platform features and containerization, see docs/CROSS_PLATFORM.md.

Quick Start on Different Platforms

Windows:

docker compose up -d

macOS/Linux:

make docker-build
docker compose up -d

Docker:

docker compose up -d  # starts the relay service

Features

OpenAI-compatible API with end-to-end encryption
Local and remote running modes
Compatibility with standard OpenAI client libraries
No token leakage to proxy servers
Cross-platform support (Windows, macOS, Linux)

Quick Start

Clone the repository
Run docker compose up to build and start the relay container
Open http://localhost:5010 in your browser to begin chatting

Development

Testing

We have comprehensive testing to ensure quality. See docs/TESTING.md for a complete overview:

# Run all Python tests
python -m pytest

# Run JS unit tests
npm run test:js

# Run crypto compatibility tests
python tests/test_crypto_compatibility_simple.py

To run every test category in one command (including Playwright and JS tests) execute TEST_COVERAGE=1 ./run_all_tests.sh (or ./run_all_tests.ps1 on Windows).

Windows PowerShell Tips

In PowerShell, use semicolons (;) for command chaining instead of ampersands (&&):

# Correct:
cd folder_path; python script.py

# Incorrect (will cause errors):
cd folder_path && python script.py

Always use explicit IPv4 addresses for reliable network testing:

# Preferred:
curl http://127.0.0.1:5010/api/v1/health

# May cause issues:
curl http://localhost:5010/api/v1/health

Contributing

We welcome contributions! See our Contributing Guide for details. Before submitting commits, install the pre-commit hooks and run them locally:

pip install pre-commit
pre-commit install
pre-commit run --all-files

These hooks run linting, tests, and spelling checks via codespell.

Security

Security is our top priority. Please report any vulnerabilities responsibly. See Security and Privacy Audit for details. token.place intentionally avoids storing user prompts or LLM responses in logs to protect user privacy.

Relay distributions now ship with Ed25519 signatures so operators can confirm binaries before running them. Verify a download with the bundled helper:

python -m utils.signing.relay_signature relay.py config/signing/relay.py.sig

Use --public-key to supply a custom key if you host your own release channel. The command exits with status code 0 on success and 1 if verification fails.

License

This project is licensed under the MIT License as detailed in LICENSE.

Name		Name	Last commit message	Last commit date
Latest commit History 3,096 Commits
.github		.github
api		api
charts/tokenplace		charts/tokenplace
clients		clients
config		config
deploy/charts/tokenplace-relay		deploy/charts/tokenplace-relay
desktop-tauri		desktop-tauri
desktop		desktop
dict		dict
docker		docker
docs		docs
hooks		hooks
integration_tests		integration_tests
k8s		k8s
outages		outages
scripts		scripts
server		server
static		static
tests		tests
types/jsencrypt		types/jsencrypt
utils		utils
.bandit		.bandit
.coveragerc		.coveragerc
.dockerignore		.dockerignore
.editorconfig		.editorconfig
.gitattributes		.gitattributes
.gitignore		.gitignore
.nvmrc		.nvmrc
.pre-commit-config.yaml		.pre-commit-config.yaml
.python-version		.python-version
.spellcheck.yaml		.spellcheck.yaml
.typos.toml		.typos.toml
AGENTS.md		AGENTS.md
CHANGELOG.md		CHANGELOG.md
CLAUDE.md		CLAUDE.md
CODE_OF_CONDUCT.md		CODE_OF_CONDUCT.md
CONTRIBUTING.md		CONTRIBUTING.md
Dockerfile		Dockerfile
LICENSE		LICENSE
Makefile		Makefile
README.md		README.md
client.py		client.py
client_simplified.py		client_simplified.py
codecov.yml		codecov.yml
config.py		config.py
docker-compose.yml		docker-compose.yml
encrypt.py		encrypt.py
eslint.config.mjs		eslint.config.mjs
llama_cpp.py		llama_cpp.py
llms.txt		llms.txt
mypy.ini		mypy.ini
package-lock.json		package-lock.json
package.json		package.json
pytest.ini		pytest.ini
relay.py		relay.py
release_metadata.py		release_metadata.py
requirements.txt		requirements.txt
run_all_tests.ps1		run_all_tests.ps1
run_all_tests.sh		run_all_tests.sh
runbook.yml		runbook.yml
server.py		server.py
tsconfig.client.json		tsconfig.client.json
tsconfig.json		tsconfig.json

Folders and files

Latest commit

History

Repository files navigation

token.place

Quickstart

Developer workflow quick reference

Key environment variables

Configuration precedence

Content moderation hooks

CI pass criteria

vision

Desktop direction

API v1 E2EE architecture baseline (v0.1.0)

Canonical entrypoints

Deployment topology

sugarkube deployment

Contents

roadmap

streaming usage

example curl request

consuming the stream in python

installation

virtual environment

unix/linux/macos

windows

install dependencies

macOS relay-only quick start

troubleshooting llama_cpp_python builds

hardware acceleration

windows

macos

unix/linux

Running the servers

Dev-like environment

Configuring relay upstream server nodes

Zero-trust relay verification

Using the Application

Relay batching

Raspberry Pi 5 deployment

Testing

Running Tests

Test Categories

Real LLM Testing

Project Documentation for LLMs

Naming Conventions

API (OpenAI-compatible)

API Endpoints

Browser CORS contract

List Models

Get Model

Chat Completions

Text Completions

Health Check

Image Generations

Community Provider Directory

Community Model Leaderboard

Community Contribution Queue

Authorising community-operated servers

Community Contribution Summary

End-to-End Encryption

System Architecture

Cross-Platform Support

Quick Start on Different Platforms

Features

Quick Start

Development

Testing

Windows PowerShell Tips

Contributing

Security

License

About

Topics

Resources

License

Code of conduct

Contributing

Uh oh!

Stars

troubleshooting `llama_cpp_python` builds

Packages