Monitoring

Monitoring

niks3 exposes Prometheus metrics at GET /metrics. The endpoint is unauthenticated, like /health; restrict it at the network/reverse-proxy layer if your server is publicly reachable.

# prometheus.yml
scrape_configs:
  - job_name: niks3
    static_configs:
      - targets: ["niks3.example.com:8080"]

Architecture note

niks3 mostly coordinates uploads rather than carrying object bytes: clients PUT directly to S3 via presigned URLs, and reads are either redirected to S3/a CDN or proxied. As a result some numbers are intentionally not exposed (e.g. compressed storage size, upload throughput) because niks3 never sees them. See the notes on individual metrics below.

Metrics

Cache inventory

Backed by a running-totals table maintained by a database trigger, refreshed into the gauges once at startup and then every minute.

Metric	Type	Description
niks3_cache_objects	gauge	Number of live objects in the cache.
niks3_cache_logical_bytes	gauge	Logical (uncompressed) size of live objects. Sourced from the client-reported NAR size, so it is not the compressed size stored in S3. NAR objects only; objects without a reported size (narinfo, logs, older clients) are excluded.

There is no metric for the compressed bucket size: S3 has no cheap bucket-size API and niks3 never sees the compressed bytes.

HTTP

Metric	Type	Labels	Description
niks3_http_requests_total	counter	method, route, status	Requests by method, matched route and status. The route label is the templated route pattern, not the raw path.
niks3_http_request_duration_seconds	histogram	method, route	Request duration. In redirect mode this measures the redirect, not the client's download from S3.
niks3_http_requests_in_flight	gauge	—	Requests currently being served.

Database

Sampled from the connection pool once a minute.

Metric	Type	Description
niks3_db_connections_in_use	gauge	Connections currently acquired from the pool.
niks3_db_connections_max	gauge	Maximum pool size.

Uploads

Metric	Type	Description
niks3_pending_closures	gauge	In-flight uploads (pending closures) not yet committed.

Garbage collection

Metric	Type	Labels	Description
niks3_gc_runs_total	counter	result	GC runs, labelled succeeded or failed.
niks3_gc_duration_seconds	histogram	—	GC run duration.
niks3_gc_objects_deleted_total	counter	—	Objects deleted from S3 and the database.
niks3_gc_last_run_timestamp_seconds	gauge	—	Unix time of the last successful GC. Held in memory, so it is absent until a GC completes after startup — alert on its absence rather than a stale value.

Runtime

The standard Go runtime (go_*) and process (process_*) collectors are also exposed.

Example queries

# HTTP error rate
sum(rate(niks3_http_requests_total{status=~"5.."}[5m]))
  / sum(rate(niks3_http_requests_total[5m]))

# 99th percentile request latency by route
histogram_quantile(0.99,
  sum by (le, route) (rate(niks3_http_request_duration_seconds_bucket[5m])))

# Connection pool saturation
niks3_db_connections_in_use / niks3_db_connections_max

# Alert: no successful GC in the last 24h (also fires if it never ran)
time() - niks3_gc_last_run_timestamp_seconds > 86400
  or absent(niks3_gc_last_run_timestamp_seconds)