Skip to content
Permalink
Branch: master
Find file Copy path
Find file Copy path
Fetching contributors…
Cannot retrieve contributors at this time
39 lines (24 sloc) 1.98 KB

Remote Read

This definition contains Prometheus statefulsets definitions set up to test remote read changes described here.

@bwplotka is using those to benchmark new remote read with Thanos sidecar on live Kubernetes cluster.

Requirements

Usage

  • Generate YAMLs from definitions:
    • make gen

You should see 2 Prometheus + Thanos definitions. One is baseline, second is a version with modified remote read that allows streaming encoded chunks.

Those resources are crafted for benchmark purposes -> they generate artificial metric data in the init container.

  • Apply baseline or improved version:
    • kubectl apply -f benchmarks/remote-read/gen-manifests/<choose>.yaml

NOTE: because of init container generating data - init can take few minutes and lots of memory (roughly 6GB per 10k series).

  • Forward gRPC sidecar port:

    • kubectl port-forward pod/<pod name> 1234:19090
  • Perform tests using test.sh (modifying parameters in script itself - heavy queries!)

    • This performs heavy queries (query all) against Thanos gRPC Store.Series of sidecar which will proxy requests as remote read to Prometheus
    • bash ./benchmarks/remote-read/test/test.sh localhost:1234
  • See resource consumption based on requests (assuming your monitor Prometheus is running on localhost:9090)

http://localhost:9090/graph?g0.range_input=1h&g0.expr=sum(container_memory_working_set_bytes%7Bpod%3D~%22prometheus.*%22%2C%20container!%3D%22%22%7D)%20by%20(container)&g0.tab=0&g1.range_input=1h&g1.expr=go_memstats_alloc_bytes%7Bpod%3D~%22prometheus.*%22%7D&g1.tab=0&g2.range_input=1h&g2.expr=sum(rate(container_cpu_user_seconds_total%7Bpod%3D~%22prometheus.*%22%2C%20container!%3D%22%22%7D%5B5m%5D))%20by%20(container)&g2.tab=0

You can’t perform that action at this time.