twopointo — server

Self-hosted Discord alternative. Phase 1 (MVP core) in progress.

Architecture (quick reference)

Client → Edge (:60420) ─── /id/* ──→ Identity (:8081)
                       └── /api/* → Official API (:8080)
                       └── /ws    → WebSocket Hub (NATS-backed)

NATS JetStream ← Official API publishes events
NATS JetStream → Edge WS Hub → WebSocket clients

• Edge (:60420) is the single entry point for all client traffic — REST and WebSocket.
  • /id/* — proxied to Identity (Authorization header forwarded, no JWT check)
  • /api/* — proxied to Official (JWT validated, X-User-ID injected, Authorization stripped)
  • /ws — WebSocket realtime gateway
• Official and Identity are internal; never call them directly from clients.
• Identity issues RS256 JWTs; Edge validates them offline via JWKS cache.
• CORS is configured via DEV_ALLOWED_ORIGINS (default: http://localhost:5173).

---

Local Dev Quickstart (fish shell)

0.5. Generate persistent identity key (once)

make identity-keygen
# Generates infra/keys/identity_rsa.pem (RSA 2048, chmod 600).
# The identity service loads this key so JWT sessions survive restarts.
# Idempotent — safe to re-run; skips if the file already exists.
# Also set IDENTITY_RSA_PRIVATE_KEY_FILE=infra/keys/identity_rsa.pem in .env
# (already the default in infra/.env.example).

1. Start infra

make dev-up
# Starts: Postgres 16, Redis 7, NATS 2.10, MinIO, runs DB migrations.

1.5. Bootstrap MinIO (one-time)

make minio-init
# Creates the storage bucket and sets the browser-upload CORS policy.
# Safe to re-run at any time (idempotent).

2. Start services (three terminals)

make run-identity   # :8081
make run-official   # :8080
make run-edge       # :60420

Env vars are loaded from .env at the repo root (copied from infra/.env.example).

3. Auth flow — all via Edge (:60420)

From P1.7 on: speak only to the Edge. Replace any :8081 or :8080 calls with :60420/id/ or :60420/api/.

# Register
curl -sf -X POST http://localhost:60420/id/auth/register \
  -H 'Content-Type: application/json' \
  -d '{"email":"alice@example.com","username":"alice","password":"S3cretPass!"}' | jq .

# Login → always returns partial_token (2FA mandatory)
set PARTIAL (curl -sf -X POST http://localhost:60420/id/auth/login \
  -H 'Content-Type: application/json' \
  -d '{"email":"alice@example.com","password":"S3cretPass!"}' | jq -r .partial_token)

# Setup 2FA (first login)
set SETUP (curl -sf -X POST http://localhost:60420/id/auth/2fa/setup \
  -H "Authorization: Bearer $PARTIAL")
set SECRET (echo $SETUP | jq -r .totp_secret)

# Get TOTP code (requires oathtool: pacman -S oath-toolkit)
set CODE (oathtool --totp --base32 $SECRET)

# Verify → full access_token
set ACCESS (curl -sf -X POST http://localhost:60420/id/auth/2fa/verify \
  -H "Authorization: Bearer $PARTIAL" \
  -H 'Content-Type: application/json' \
  -d "{\"code\":\"$CODE\"}" | jq -r .access_token)

4. Core REST (all via Edge /api/*)

# Create server
set SID (curl -sf -X POST http://localhost:60420/api/servers \
  -H "Authorization: Bearer $ACCESS" \
  -H 'Content-Type: application/json' \
  -d '{"name":"My Server"}' | jq -r .id)

# My servers
curl -sf http://localhost:60420/api/me/servers \
  -H "Authorization: Bearer $ACCESS" | jq .

# Create channel
set CID (curl -sf -X POST http://localhost:60420/api/servers/$SID/channels \
  -H "Authorization: Bearer $ACCESS" \
  -H 'Content-Type: application/json' \
  -d '{"name":"general"}' | jq -r .id)

# Send message
set MID (curl -sf -X POST http://localhost:60420/api/channels/$CID/messages \
  -H "Authorization: Bearer $ACCESS" \
  -H 'Content-Type: application/json' \
  -d '{"content":"Hello!"}' | jq -r .id)

# List messages (newest first)
curl -sf "http://localhost:60420/api/channels/$CID/messages" \
  -H "Authorization: Bearer $ACCESS" | jq '.messages[].content'

# Delete message (204 No Content)
curl -sf -X DELETE http://localhost:60420/api/messages/$MID \
  -H "Authorization: Bearer $ACCESS" -o /dev/null -w "%{http_code}\n"

5. Web Dev Client (optional, P1.7)

make web-install   # bun install (first time)
make web-dev       # starts Vite on http://localhost:5173

The web client speaks only to the Edge. CORS is pre-configured for http://localhost:5173.

5. WebSocket realtime (requires websocat)

# Connect (token in query param — standard for WS clients)
websocat "ws://localhost:60420/ws?token=$ACCESS"
# ← {"op":"HELLO","d":{"heartbeat_interval":30000}}

# Subscribe (send after connect):
# {"op":"SUBSCRIBE","d":{"server_id":"<SID>","channel_ids":["<CID>"]}}
# ← {"op":"EVENT","t":"PRESENCE_UPDATE","d":{"status":"online",...}}

# Now POST a message in another terminal → WS receives MESSAGE_CREATE event
# POST /channels/:id/typing → WS receives TYPING_START event
# DELETE /messages/:id     → WS receives MESSAGE_DELETE event

See docs/DEV_SMOKE_TESTS.md for the full step-by-step checklist including exact expected payloads.

---

Make targets

Target	Description
make identity-keygen	Generate RSA key for identity service (once, idempotent)
make dev-up	Start infra (Postgres + Redis + NATS + MinIO + migrations)
make minio-init	Create MinIO bucket + CORS policy (run once after make dev-up)
make dev-down	Stop infra
make dev-up-all	Start infra + app services via Docker build
make run-identity	Run identity service locally (reads .env)
make run-official	Run official API locally
make run-edge	Run edge proxy + WS hub locally
make build-all	Build all three binaries to bin/
make gen-permissions	Regenerate Go + TS permission constants from permissions.json
make dev-delete-user EMAIL=x@y.z	Delete one user + all their data from the local dev DB (irreversible)
make dev-reset-db	Truncate every app table (prompts for confirmation; schema/migrations untouched)
make test	Run all tests

---

Ports

Service	Port	Notes
Edge	60420	Public-facing (REST + WebSocket)
Official API	8080	Internal only
Identity	8081	Internal only
PostgreSQL	5432
Redis	6379
NATS	4222	JetStream enabled
MinIO	9000 / 9001	S3-compat API / console

---

Troubleshooting

Port conflicts or wrong Docker context after reboot

dev-up.sh handles both automatically:

1. Wrong Docker context (desktop-linux instead of default) — causes /host_mnt mount failures and compose errors. The script detects and aborts with a clear fix hint:

# Auto-fix (switches context + restarts Docker daemon if systemd-managed):
./scripts/dev-up.sh --force
# Manual fix:
docker context use default

2. Busy infra ports (e.g. port 4222 held by another NATS instance) — the script auto-remaps to alternative host ports instead of failing:

[info]  Port 4222 (NATS client): held by 'nats-server' (pid 1234) → remapping to 14222
[info]  Port 9000 (MinIO API):   held by Docker container 'old-minio' → remapping to 19000

  ── Active port remaps ──────────────────────────────────────────
  nats  4222 → 14222  NATS_URL updated
  minio 9000 → 19000  MINIO_ENDPOINT / STORAGE_ENDPOINT updated
  ────────────────────────────────────────────────────────────────

Remaps are host-port only — internal Docker networking (nats:4222 etc.) is unchanged. Go services in this run inherit the updated env vars automatically. The generated override file (infra/docker-compose.override.autogen.yml) is removed by dev-down.sh.

Port already in use after crash / manual Ctrl-C

# Force-kill any leftover twopointo process on ports 8081/8080/60420/5173,
# then start fresh:
./scripts/dev-up.sh --force      # bash
fish scripts/dev-up.fish --force # fish

# Or just clear ports without starting:
./scripts/dev-down.sh --force
fish scripts/dev-down.fish --force

Only kills processes whose cmdline matches identity, official, edge, or vite/node. Unrelated processes on the same port are skipped with a warning.

2FA broken or account stuck — delete a user

make dev-delete-user EMAIL=alice@example.com

Deletes the user and all their data (servers, messages, assets, 2FA, refresh tokens) in one transaction. After deletion, register the same address again via the web UI or curl.

Start completely fresh — wipe the DB

make dev-reset-db
# Type RESET when prompted

Truncates every app table. The schema stays intact — no make migrate-up needed. All services can keep running; just re-register users as usual.

Sessions lost / "Session expired" after restarting identity service

Identity uses an RSA key to sign JWTs. Without a persistent key file, a new ephemeral key is generated on every startup, invalidating all existing tokens.

Fix (one-time):

make identity-keygen
# Then ensure your .env contains:
# IDENTITY_RSA_PRIVATE_KEY_FILE=infra/keys/identity_rsa.pem

After this, restarting identity no longer invalidates sessions.

2FA setup screen appears on every login

Migration 006 (recovery_confirmed_at column) has not been applied:

make migrate-up

Then complete one login + confirm recovery codes; subsequent logins will go straight to the TOTP entry screen.