A high-throughput Nostr analytics backend for video sharing apps, built with Rust and ClickHouse.
Funnel is the analytics and search layer for a Vine-style video sharing app built on Nostr. It ingests events from an external relay and provides:
- Video stats — reaction counts, comment counts, reposts
- Search — find videos by hashtag or content
- Custom feeds — trending videos, sorted by engagement, filtered by author
- Analytics — DAU/WAU/MAU, top creators, popular hashtags
┌────────────────────┐
│ Nostr Clients │
└─────────┬──────────┘
│
┌─────────────────┴─────────────────┐
│ │
│ Nostr protocol │ HTTP
│ (EVENT/REQ/CLOSE) │ (stats, search, feeds)
▼ ▼
┌───────────────┐ ┌───────────────┐
│ External Relay│ │ REST API │
│ │ │ (Rust) │
└───────┬───────┘ └───────┬───────┘
│ │
│ WebSocket │ queries
▼ ▼
┌───────────────┐ ┌───────────────┐
│ Ingestion │──── writes ──────▶│ ClickHouse │
│ Service │ │ │
└───────────────┘ └───────────────┘
External Relay handles EVENT/REQ/CLOSE, subscriptions, and primary storage.
Ingestion subscribes to the relay via WebSocket and streams all events to ClickHouse with batched inserts.
ClickHouse stores events for complex queries, aggregations, and analytics that Nostr REQ doesn't support.
REST API exposes video stats, search, and feeds to the app.
Standard Nostr queries are great for real-time protocol operations, but we need:
- Aggregations — count reactions, comments, reposts (Nostr REQ doesn't support)
- Custom sort orders — trending, popular (beyond
created_at) - Full-text search — across titles and content
- Analytics — DAU/WAU/MAU, creator stats, hashtag trends
- Data exports — for recommendation systems
ClickHouse excels at these analytical queries.
| Endpoint | Description |
|---|---|
GET /health |
Health check |
GET /metrics |
Prometheus metrics |
GET /api/videos/{id}/stats |
Get reaction, comment, and repost counts for a video |
GET /api/videos?sort=recent|trending&limit= |
List videos with custom sort |
GET /api/users/{pubkey}/videos?limit= |
Get videos by a specific creator |
GET /api/search?tag=...&q=...&limit= |
Search by hashtag or text |
GET /api/stats |
Total event and video counts |
All endpoints return JSON with Cache-Control headers.
- Docker and Docker Compose
- ClickHouse instance (self-hosted or ClickHouse Cloud)
- An external Nostr relay to ingest events from
Apply the schema to your ClickHouse instance:
clickhouse-client --multiquery < docs/schema.sqlOr for ClickHouse Cloud:
clickhouse-client \
--host your-host.clickhouse.cloud \
--secure \
--user default \
--password your-password \
--multiquery < docs/schema.sqlcp .env.example .env
# Edit .env with your settings:
# RELAY_URL=wss://your-relay.example.com
# CLICKHOUSE_URL=https://host:8443?user=default&password=xxxdocker compose up -dThis starts:
- API on port 8080 (REST endpoints)
- Prometheus on port 9090 (metrics)
- Ingestion (internal, streams events from relay to ClickHouse)
The ingestion service operates in two modes:
Automatically starts with docker compose up and streams new events in real-time:
docker compose up -d
docker compose logs -f ingestionLive mode subscribes from the last known event timestamp (with a 2-day buffer) so it catches up on any events missed while stopped.
To import all historical events from a relay, run the backfill container:
# Run backfill (one-time, will exit when complete)
docker compose run --rm backfillBackfill paginates through the entire relay history in batches of 5,000 events, walking backwards in time. Progress is logged:
INFO Fetching batch until=2024-01-15T10:30:00Z limit=5000 total_so_far=150000
INFO Received batch count=5000 oldest=2024-01-14T22:15:33Z
INFO Inserted batch_inserted=5000 total_events=155000
Notes:
- Backfill is safe to re-run — ClickHouse deduplicates by event ID
- Run backfill in
tmuxorscreenfor long-running syncs - Stop early with
Ctrl+Cif needed; progress is saved to ClickHouse - Live ingestion and backfill can run simultaneously
| Variable | Required | Default | Description |
|---|---|---|---|
RELAY_URL |
Yes | — | WebSocket URL of the Nostr relay to ingest from |
CLICKHOUSE_URL |
Yes | — | ClickHouse server URL (e.g., https://host:8443) |
CLICKHOUSE_USER |
No | default |
ClickHouse username |
CLICKHOUSE_PASSWORD |
Yes | — | ClickHouse password |
CLICKHOUSE_DATABASE |
No | nostr |
ClickHouse database name |
BATCH_SIZE |
No | 1000 |
Events per insert batch |
BACKFILL |
No | — | Set to 1 to run in backfill mode |
RUST_LOG |
No | info |
Log level (debug, info, warn, error) |
RELAY_URL=wss://relay.example.com
CLICKHOUSE_URL=https://your-instance.clickhouse.cloud:8443
CLICKHOUSE_USER=default
CLICKHOUSE_PASSWORD=your-password
CLICKHOUSE_DATABASE=nostrcargo build --releasecargo test# Start prometheus for metrics
docker compose up -d prometheus
# Run ingestion service
RELAY_URL=wss://relay.example.com \
CLICKHOUSE_URL=http://localhost:8123 \
cargo run --bin funnel-ingestion
# Run API server
CLICKHOUSE_URL=http://localhost:8123 \
cargo run --bin funnel-apijust build # Build all crates
just test # Run tests
just fmt # Format code
just lint # Run clippy
just up # Start all services
just down # Stop all servicescrates/
├── proto/ # Nostr types, video event parsing
├── clickhouse/ # ClickHouse client and queries
├── ingestion/ # WebSocket subscriber, batch processor
├── api/ # Axum REST API
└── observability/# Tracing and metrics
docs/
├── plan.md # Implementation roadmap
├── schema.sql # ClickHouse schema
└── deployment.md # Production deployment guide
config/
└── prometheus.yml# Prometheus scrape config
Funnel indexes video events per NIP-71:
- Kind 34235 — Normal videos
- Kind 34236 — Short videos (vertical format)
Both are addressable/replaceable events identified by the d tag.
docs/plan.md— Implementation plan and architecturedocs/schema.sql— ClickHouse schema with all tables and viewsdocs/deployment.md— Production deployment with Ansible