cache: harden helper cache tiers and discovery

[szibis]

Summary

• move helper/read caches onto canonicalized keys so Grafana refreshes can reuse the same entries across param ordering and alias variants
• harden metadata and detected-field discovery so successful empty strict queries stay strict instead of broadening into relaxed helper/discovery data
• export tiered cache visibility through both /metrics and OTLP, then add matching dashboard panels for per-tier requests, hits, stale serves, backend fallthrough, object counts, and bytes
• keep helper stale fallback local-first while preserving Loki-facing and VictoriaLogs-facing compatibility
• opportunistically compress peer write-through pushes only after the remote peer has advertised support on existing cache GET/hot responses, and accept compressed peer cache POST bodies without breaking mixed-version fleets
• request compressed peer snapshot warm responses and skip periodic snapshot rewrites when only the saved-at timestamp changed, trimming east-west transfer and avoidable disk churn on patterns and label-values persistence

Validation

• go test ./internal/cache
• go test ./internal/proxy
• go test ./internal/metrics
• python3 scripts/ci/check_changelog_pr.py --base origin/main --head HEAD
• jq empty dashboard/loki-vl-proxy.json charts/loki-vl-proxy/dashboards/loki-vl-proxy.json

[github-actions]

PR Quality Report

Compared against base branch main.

Coverage and tests

Signal	Base	PR	Delta
Test count	1794	1807	13
Coverage	89.9%	89.3%	-0.7% (regressed)

Compatibility

Track	Base	PR	Delta
Loki API	100.0%	11/11 (100.0%)	0.0% (stable)
Logs Drilldown	100.0%	17/17 (100.0%)	0.0% (stable)
VictoriaLogs	100.0%	11/11 (100.0%)	0.0% (stable)

Compatibility components

Track	Component	Base	PR	Delta
Loki API	label_values	2/2 (100.0%)	2/2 (100.0%)	0.0% (stable)
Loki API	labels	4/4 (100.0%)	4/4 (100.0%)	0.0% (stable)
Loki API	metrics	2/2 (100.0%)	2/2 (100.0%)	0.0% (stable)
Loki API	otel	1/1 (100.0%)	1/1 (100.0%)	0.0% (stable)
Loki API	query_range	1/1 (100.0%)	1/1 (100.0%)	0.0% (stable)
Loki API	series	1/1 (100.0%)	1/1 (100.0%)	0.0% (stable)
Logs Drilldown	detected_fields	11/11 (100.0%)	11/11 (100.0%)	0.0% (stable)
Logs Drilldown	label_values	1/1 (100.0%)	1/1 (100.0%)	0.0% (stable)
Logs Drilldown	level_volume	2/2 (100.0%)	2/2 (100.0%)	0.0% (stable)
Logs Drilldown	patterns	1/1 (100.0%)	1/1 (100.0%)	0.0% (stable)
Logs Drilldown	service_logs	1/1 (100.0%)	1/1 (100.0%)	0.0% (stable)
Logs Drilldown	service_selection	1/1 (100.0%)	1/1 (100.0%)	0.0% (stable)
VictoriaLogs	detected_fields	4/4 (100.0%)	4/4 (100.0%)	0.0% (stable)
VictoriaLogs	field_values	3/3 (100.0%)	3/3 (100.0%)	0.0% (stable)
VictoriaLogs	index_stats	1/1 (100.0%)	1/1 (100.0%)	0.0% (stable)
VictoriaLogs	stream_translation	1/1 (100.0%)	1/1 (100.0%)	0.0% (stable)
VictoriaLogs	synthetic_labels	1/1 (100.0%)	1/1 (100.0%)	0.0% (stable)
VictoriaLogs	volume_range	1/1 (100.0%)	1/1 (100.0%)	0.0% (stable)

Performance smoke

Lower CPU cost (ns/op) is better. Lower benchmark memory cost (B/op, allocs/op) is better. Higher throughput is better. Lower load-test memory growth is better. Benchmark rows are medians from repeated samples.

Signal	Base	PR	Delta
QueryRange cache-hit CPU cost	1279.0 ns/op	1422.0 ns/op	+11.2% (stable)
QueryRange cache-hit memory	200.0 B/op	200.0 B/op	0.0% (stable)
QueryRange cache-hit allocations	7.0 allocs/op	7.0 allocs/op	0.0% (stable)
QueryRange cache-bypass CPU cost	1622.0 ns/op	1779.0 ns/op	+9.7% (stable)
QueryRange cache-bypass memory	271.0 B/op	278.0 B/op	+2.6% (stable)
QueryRange cache-bypass allocations	7.0 allocs/op	7.0 allocs/op	0.0% (stable)
Labels cache-hit CPU cost	632.0 ns/op	693.3 ns/op	+9.7% (stable)
Labels cache-hit memory	56.0 B/op	48.0 B/op	-14.3% (stable)
Labels cache-hit allocations	4.0 allocs/op	3.0 allocs/op	-25.0% (improved)
Labels cache-bypass CPU cost	824.4 ns/op	872.7 ns/op	+5.9% (stable)
Labels cache-bypass memory	84.0 B/op	53.0 B/op	-36.9% (stable)
Labels cache-bypass allocations	4.0 allocs/op	3.0 allocs/op	-25.0% (improved)
High-concurrency throughput	125196.0 req/s	109538.0 req/s	-12.5% (stable)
High-concurrency memory growth	0.8 MB	0.4 MB	-50.0% (stable)

State

• Coverage, compatibility, and sampled performance are reported here from the same PR workflow.
• This is a delta report, not a release gate by itself. Required checks still decide merge safety.
• Performance is a smoke comparison, not a full benchmark lab run.
• Delta states use the same noise guards as the quality gate (percent + absolute + low-baseline checks), so report labels match merge-gate behavior.