feat: expose a native Thanos StoreAPI to serve long-term metrics from PostgreSQL#43
Merged
Conversation
Introduce configuration for the upcoming Thanos StoreAPI gRPC server: - GRPC_ADDR (default :10901) for the gRPC listen address - STORE_API_EXTERNAL_LABELS for the external labels advertised to Thanos via the StoreAPI Info call (used for deduplication) Issue: ARTESCA-17670
Add a new storeapi package with the conversion that maps a Thanos StoreAPI SeriesRequest (matchers + time range) to a prompb.Query, so the existing SQL query builder can be reused unchanged for the StoreAPI Series path. This introduces the github.com/thanos-io/thanos/pkg/store/storepb dependency (thanos v0.41.0, compatible with the current prometheus/prometheus v0.312.0). Issue: ARTESCA-17670
eg-ayoub
reviewed
Jun 18, 2026
eg-ayoub
reviewed
Jun 18, 2026
Add BuildSeries, which groups database rows into Thanos StoreAPI series: - merges the metric name, stored labels and configured external labels into a sorted label set (external labels take precedence on collision, like the Thanos sidecar, so the StoreAPI output matches the sidecar path) - sorts series by label set as required by the StoreAPI - encodes samples into XOR chunks (120 samples per chunk) Issue: ARTESCA-17670
Add the read queries the Thanos StoreAPI needs: - MinSampleTimestamp: oldest metric_time, used to advertise the store's min time (with a boolean for the empty-database case) - LabelNames / LabelValues: distinct label names and values (supersets, matchers/time not applied, which the StoreAPI permits) Issue: ARTESCA-17670
Add the gRPC server backing the StoreAPI, implementing both storepb.StoreServer (Series, LabelNames, LabelValues) and infopb.InfoServer (Info): - Info advertises the configured external labels and a time range whose min is the oldest sample in the database (MaxInt64 when empty, so Thanos skips it) - Series converts the request to SQL, queries the database, builds XOR-chunked series and streams them - LabelNames/LabelValues expose the stored labels plus the external labels Issue: ARTESCA-17670
TeddyAndrieux
force-pushed
the
feat/thanos-store-api
branch
from
June 18, 2026 09:24
1b69af4 to
b4fb47c
Compare
eg-ayoub
approved these changes
Jun 18, 2026
Wire the Thanos Store/Info gRPC server into the DI container and start it from main on GRPC_ADDR (default :10901), next to the existing HTTP server. Issue: ARTESCA-17670
Describe the gRPC StoreAPI exposed by the adapter: the GRPC_ADDR and STORE_API_EXTERNAL_LABELS settings, how to register it as a Thanos Query endpoint, the external-labels behaviour, and a Kubernetes Service example. Issue: ARTESCA-17670
Adopt github.com/scality/go-errors (the error library used by scality/file-reflector) for consistency across Scality Go repositories. Each failure now wraps a package-level ErrXxx sentinel - the error category, so errors.Is keeps matching through the chain - via errors.Wrap(ErrXxx, errors.CausedBy(rawErr), errors.WithProperty(...)), with go-errors imported unaliased as errors. Sentinel titles preserve the previous messages. Two use-case tests that built fake errors switch to the standard-library errors package.
…flector Mirror scality/file-reflector's documentation set for consistency across Scality Go repositories: - DESIGN.md: architecture (layers, ingestion pipeline, read paths, storage model) and the design decisions (PostgreSQL store, native Thanos StoreAPI, external labels, XOR chunks, single-replica rule) - CONTRIBUTING.md: layering, coding conventions (slog, go-errors, testify), commit and PR workflow - CLAUDE.md: short working guide importing the doc set - README.md: reworked to file-reflector's structure (badges, how it works, setup, configuration table, container image, development), fixing stale environment variables (HTTP_ADDR, POSTGRESQL_DATABASE_NAME, LOGGER_LOG_LEVEL) and Docker build args
Adopt the same build/CI conventions as scality/file-reflector for consistency across Scality Go repositories: - workflows: replace the previous set with build / pre-merge / post-merge / promote / release / review (reusable build via workflow_call, GHCR images, git-describe tags); drop the file-reflector-specific capability and mockery jobs - Dockerfile: golang:1.26 + distroless/static:nonroot (USER 65532), VERSION build-arg, build ./cmd; drop the file-reflector-specific setcap step - add .dockerignore, align .golangci.yaml (v2) and .gitignore - move CODEOWNERS under .github/, switch renovate to .github/renovate.json5 - fix the lint issues the new golangci config surfaces (errcheck, wsl_v5)
Give debug-level visibility into the queries flowing through the adapter, on both edges: - the StoreAPI server logs every request it receives (Series, LabelNames, LabelValues) with the raw matchers - including external-label matchers before they are stripped - and the requested time range; - the remote-read use case logs each received Prometheus query; - the PostgreSQL client logs every statement it runs - reads, label inserts, label name/value lookups, the startup metric recovery, and value COPYs - with its arguments. All of these use log/slog at debug level and stay silent at the default info level.
The empty-string __name__ equality branch built "(l.metric_name IS NULL OR name = '')", but `name` is not a column of either queried table (it is `metric_name` in `metric_labels`), so PostgreSQL rejects the query with `column "name" does not exist` whenever a `__name__=""` matcher reaches the builder. Qualify it as `l.metric_name`. This branch is now reachable from Thanos Query through the StoreAPI, not only via remote_read.
TeddyAndrieux
force-pushed
the
feat/thanos-store-api
branch
from
June 18, 2026 09:55
b4fb47c to
ce0a29a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The adapter now exposes a native Thanos StoreAPI (gRPC) so a Thanos Query can
federate the PostgreSQL-stored long-term metrics directly, without a Prometheus
remote_readpassthrough behind a sidecar.A Thanos sidecar only advertises its Prometheus' local TSDB time range, so it
cannot surface the history a Prometheus serves through
remote_readto thisadapter. Exposing the StoreAPI from the adapter makes the PostgreSQL data a
first-class Thanos store, queryable for the whole range it actually holds.
See
DESIGN.mdfor the full reasoning.Changes
pkg/presentation/storeapiimplementingstorepb.StoreServer(Series / LabelNames / LabelValues) andinfopb.InfoServer(Info). Samples are streamed as XOR chunks; series aresorted; external labels are configured via
STORE_API_EXTERNAL_LABELSandapplied to every series (with precedence on name collision), exactly like a
sidecar applies a Prometheus' external labels. Matchers on an external label
are validated and stripped before hitting SQL.
GRPC_ADDR, default:10901), with graceful shutdown and the standardgrpc_health_v1healthservice for Kubernetes probes.
github.com/pkg/errorstogithub.com/scality/go-errors(package-levelErrXxxsentinels +Wrap).(distroless, statically linked, non-root),
.golangci.yaml(v2), renovate,CODEOWNERS, ignore files.
DESIGN.md,CONTRIBUTING.md,CLAUDE.md, and rework theREADME.md.and the SQL executed against PostgreSQL, at debug level.
Testing
go test ./...- greengolangci-lint run- 0 issuesregisters and advertises its oldest sample, and a metric returns its full
history through Thanos (external-label filtering included).
Issue: ARTESCA-17670