You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I have HPA enabled on my cortex-tenant deployment, but after scale, other pod doesn't receive timeseries.
When I check"cortex_tenant_timeseries_received" metrics only the first pod has count
But when I check cortex_tenant_timeseries_batches_received metrics, other pod receive some batch but no much like the first pod :
Result series: 3
pod 1 : 25353
pod 2: 4
pod 3 : 15
Maybe we can add an nginx proxy (or other app) to loadbalance the query between the backend ?
Tell me and I can try to implement it.
The text was updated successfully, but these errors were encountered:
maybe the problem come from prometheus remote write process
I did some research and the problem is indeed prometheus shards (see queue_config).
Prometheus uses an auto-sizing algorithm for the number of shards, and it's entirely possible that the number of shards is 1, so only 1 cortex-tenant pod is used.
I have HPA enabled on my cortex-tenant deployment, but after scale, other pod doesn't receive timeseries.
When I check"cortex_tenant_timeseries_received" metrics only the first pod has count
But when I check cortex_tenant_timeseries_batches_received metrics, other pod receive some batch but no much like the first pod :
Result series: 3
pod 1 : 25353
pod 2: 4
pod 3 : 15
Maybe we can add an nginx proxy (or other app) to loadbalance the query between the backend ?
Tell me and I can try to implement it.
The text was updated successfully, but these errors were encountered: