Observability

可观测性 / Observability

SPACK 被设计为一个可运营的静态资源运行时，而不只是一个文件服务器。它默认暴露健康检查、Prometheus 指标、Go runtime 指标、debug runtime 和 Grafana dashboard。

SPACK is designed as an operable static asset runtime, not just a file server. It exposes health checks, Prometheus metrics, Go runtime metrics, a debug runtime, and Grafana dashboards.

Endpoint Overview

Endpoint	Runtime	说明 / Description
/healthz	main HTTP	综合健康检查 / Aggregate health check
/livez	main HTTP	liveness
/readyz	main HTTP	readiness
/catalog	main HTTP	当前 catalog 快照 / Current catalog snapshot
/robots.txt	main HTTP	静态或生成的 robots / Static or generated robots
/prometheus	debug HTTP	Prometheus metrics
/debug/statsviz	debug HTTP	Go runtime visualization
/pprof/*	debug HTTP	pprof endpoints

默认端口：

• main HTTP: 80
• debug runtime: 8080

Default ports:

• main HTTP: 80
• debug runtime: 8080

指标结构 / Metrics Families

flowchart TB
    Metrics["/prometheus"] --> HTTP["HTTP request metrics"]
    Metrics --> Delivery["asset delivery metrics"]
    Metrics --> Resolver["resolver metrics"]
    Metrics --> Pipeline["pipeline and stage metrics"]
    Metrics --> Catalog["catalog gauges"]
    Metrics --> Cache["asset cache metrics"]
    Metrics --> Task["task and scheduler metrics"]
    Metrics --> Async["async concurrency metrics"]
    Metrics --> DIX["dix lifecycle metrics"]
    Metrics --> Runtime["Go and process metrics"]
    Metrics --> Info["build/config/runtime info"]

Loading

主要指标：

• spack_http_requests_in_flight
• spack_pipeline_stage_runs_total
• spack_pipeline_stage_duration_seconds
• spack_pipeline_variants_generated_total
• spack_catalog_assets_current
• spack_catalog_variants_current
• spack_asset_cache_*
• spack_task_runs_total
• spack_source_rescan_*
• spack_artifact_janitor_*
• spack_cache_warmer_*
• spack_task_scheduler_*
• spack_async_*
• spack_dix_*
• spack_build_info
• spack_config_info
• spack_runtime_start_time_seconds

Representative metrics:

• spack_http_requests_in_flight
• spack_pipeline_stage_runs_total
• spack_pipeline_stage_duration_seconds
• spack_pipeline_variants_generated_total
• spack_catalog_assets_current
• spack_catalog_variants_current
• spack_asset_cache_*
• spack_task_runs_total
• spack_source_rescan_*
• spack_artifact_janitor_*
• spack_cache_warmer_*
• spack_task_scheduler_*
• spack_async_*
• spack_dix_*
• spack_build_info
• spack_config_info
• spack_runtime_start_time_seconds

Grafana Dashboards

仓库内包含两个 dashboard JSON：

The repository includes two dashboard JSON files:

• deploy/grafana/spack-overview-dashboard.json
• deploy/grafana/spack-fleet-instance-dashboard.json

用途：

• overview dashboard: 单实例或单服务视角。
• fleet dashboard: 多实例 fleet 视角，带 instance 变量。

Use cases:

• overview dashboard: single-instance or single-service view.
• fleet dashboard: multi-instance fleet view with an instance variable.

运行时信息 / Runtime Metadata

spack_build_info 暴露低基数构建标签：

• app version
• Go version
• VCS revision

spack_config_info 暴露低基数运行配置：

• asset backend
• compression mode
• memory-cache state
• frontend hint/cache state
• robots state
• image state
• logger level

spack_runtime_start_time_seconds 用于关联重启和部署。

spack_runtime_start_time_seconds is useful for correlating restarts and deployments.

日志 / Logs

请求日志会包含 delivery mode：

Request logs include delivery mode:

• memory_cache_hit
• memory_cache_fill
• sendfile
• sendfile_range

调试时建议临时开启：

For debugging, temporarily enable:

SPACK_LOGGER_LEVEL=debug
SPACK_HTTP_REQUEST_LOG_DETAIL=true

生产环境建议：

• 默认不要暴露 debug runtime 到公网。
• /prometheus 放在受信任网络或 Prometheus scrape path。
• pprof 只在排障窗口开启访问。

Production recommendations:

• Do not expose the debug runtime publicly by default.
• Keep /prometheus in a trusted network or Prometheus scrape path.
• Allow pprof access only during debugging windows.

运维排障 / Troubleshooting

现象 / Symptom	检查 / Check
请求返回 identity 而不是 br/zstd/gzip	确认 Accept-Encoding、压缩模式、sidecar 或 lazy 生成完成。Check Accept-Encoding, compression mode, sidecar availability, or lazy completion.
SPA route 404	确认 assets.fallback.target 存在于 source root。Verify fallback target exists in source root.
/robots.txt 不是生成内容	检查 source root 是否已有静态 robots.txt，以及 robots.override。Check static robots.txt and robots.override.
CPU 高	检查 pipeline stage metrics、k6 压测、pprof。Check pipeline stage metrics, k6 load, and pprof.
catalog 未更新	检查 source rescan metrics 和 fsnotify 事件。Check source rescan metrics and fsnotify events.