using thanos for querying?

[ryandawsonuk]

I'm currently querying against the prometheus of the cluster monitoring operator, hitting the prometheus-k8s service. That works fine. Now I've a situation where my queries could easily exceed the 11,000 data point max resolution on prometheus. I'm guessing that limit applies for the cluster monitoring operator's prometheus too. So I'm wondering about Thanos. I see there's references to Thanos in this codebase. Is Thanos automatically installed with the operator and is there a Thanos query API available for me to query? Or is Thanos something that I would have to install separately?

[simonpasquier]

You can use the Thanos querier service but you'll hit the same limit. If you're returning that many datapoints, you have to split the query in smaller intervals from client-side.

[ryandawsonuk]

Probably I've not explained my aim well or maybe I've misunderstood what the downsampling feature in Thanos does.

I want to query kube-state-metrics pod data over a time range of about a month with intervals of days or weeks. Each interval would have more than 11,000 data points if the raw prometheus data were used. But I'm thinking that with Thanos I could have access to downsampled data. I believe Thanos would be automatically deriving alternative time series data with a resolution which is as though the scrapes had taken place at 5m and 1hr intervals. For my case if I could use the 1hr-resolution downsampled data my queries should be fine. I believe the concept of auto-downsampling means it automatically chooses between raw, 5m or 1hr resolutions.

So I'm hoping I can keep the interval but change which sample resolution the query is run against. Apologies for not explaining that. Please do let me know if there's anything I might be misunderstanding or here or any red flags for doing this cluster monitoring (e.g. cluster monitoring retention periods might be different or configuring them might be different).

[s-urbaniak]

@ryandawsonuk downsampling is part of Thanos Compactor which is not deployed in the in-cluster monitoring stack. We only deploy Thanos Querier, Thanos Sidecar, and Thanos Ruler (for user workload monitoring).

[ryandawsonuk]

Thanks for letting me know @s-urbaniak . I think that answers my key question.