[docs][proposal] Scaled Prometheus Pipeline #2118
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Scott8440
force-pushed
the
thanosProposal
branch
from
47f47ae
to
20a0162
Compare
Scott8440
force-pushed
the
thanosProposal
branch
from
20a0162
to
80f6d03
Compare
xjtian
suggested changes
Jul 30, 2020
There was a problem hiding this comment.
Reorganize the doc a bit for me please:
- Problem statement: as-is
- Solution proposal: Thanos with single prom server (or possibly HA) and object storage. Elaborate on the current proposed architecture on the push and query sides - what will you deploy, what talks to what, data flow for timeseries from the edge. Describe what object storage is and what our options for it are for public cloud and on-prem deployments
- Implementation details: query-side
- Implementation details: push-side
|
|
||
| Object storage will allow us to only store a few hours of metrics on the server itself (potentially keeping everything in-memory) and then exporting older metrics to object storage elsewhere. For example on an AWS deployment metrics would be stored in S3. | ||
|
|
||
| We will then deploy multiple Querier components behind a load balancer which are configured to talk to both the prometheus server and the Object storage. This will distribute the compute and I/O load away from the prometheus server to the stateless querier components which can be trivially scaled horizontally to handle increase query loads. |
|
|
||
| With this setup, we only need to deploy the Thanos `sidecar` and multiple `Querier` components, along with Object storage to achieve faster queries. | ||
|
|
||
| ### Cortex |
There was a problem hiding this comment.
let's just keep this doc about thanos which seems like a far more approriate solution to our problem space
Signed-off-by: Scott8440 <scott8440@gmail.com>
xjtian
approved these changes
Aug 3, 2020
|
|
||
| |Step |Est. Time | | ||
| |--- |--- | | ||
| |Deploy Thanos locally and experiment with loads to validate query time improvements |2 wk | |
There was a problem hiding this comment.
you'll probably have to deploy this on AWS yourself otherwise you won't be able to test what kind of impact object storage has on query performance
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Design doc discussing how to scale prometheus with the end goal of improving query times and supporting increased capacity.
Test Plan
N/a