Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add store-gateway components (grafana/phlare#701)
This PR adds a store-gateway component similar to what exists in Mimir. https://github.com/grafana/mimir/blob/main/pkg/storegateway/gateway.go This is still very early, so far I've decided to depends on Mimir to benefits from the shuffle sharding and replication strategy. I'm not planning to provide any block persistence for now only in-memory. In the future we should use memcached for symbols and tsdb index. The store gateway open block for each tenant within 24h - (now -2h). Since we don't have a compactor a single gateway can download data duplicated by the replication factor of ingesters. This means we need to deduplicate blocks data while streaming now. To speed this up I've implemented a BufferedIterator that helps merging multiple iterator in parallel. In the future we should probably only download compacted blocks. For now we always deduplicate even in the ingester code even though no duplication happens there, the PR was big enough and I don't think it's a big concern right now. The store-gateway also replicate data for high availability which means more duplicates are sent to queriers. We should consider sending block ULID and request querier to select the one to consider. To keep thing simple for the query path right now, we split the queries using the queryStoreAfter configuration. Ultimately if we dedupe by blocks we should be able to remove that configuration and select blocks that needs to be querier from the querier directly. --------- Co-authored-by: Christian Simon <simon@swine.de>
- Loading branch information