Skip to content

[Parquet] Store Gateway Hybrid Mode #7140

@yeya24

Description

@yeya24

Is your feature request related to a problem? Please describe.
#7046 introduces a new mode for store gateway to query parquet files intead of TSDB blocks. However, for production environment where store gateway still queries TSDB blocks, it is not easy to migrate to parquet mode without down time.

Besides, Parquet files are converted from TSDB blocks and it takes longer time for them to be available. Before parquet files are converted, to make latest data queryable Store Gateway might still need to support querying TSDB blocks.

Describe the solution you'd like
I want to propose a hybrid mode of Store Gateway that is able to query both parquet files and TSDB blocks. Hybrid Store Gateway still needs to sync TSDB blocks at start up and periodically but it only needs to sync most recent blocks for a configurable time period (such as recent X days). For the blocks to query asked from Querier, if a block is available as parquet file then parquet file is queried otherwise, TSDB block is queried. This hybrid mode allows smooth migration from the TSDB store gateway to full parquet gateway eventually.

This is a very similar approach to what we did for parquet querier. If a block is available as parquet file then querier queries parquet file otherwise it queries Store Gateway for TSDB blocks.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions