End to end integration testing environment and benchmarking system

[gouthamve]

Quoting Julius:

We're not really excellent in the correctness / bug-free-ness department yet, partially because certain key components either lack tests or you'd have to run them in a real-world scenario for a while to discover certain bugs. I'm especially looking at our under-tested service discovery mechanisms here that require e.g. a Zookeeper or Consul as a dependency to test them for real. It'd be cool to have a test environment that runs a Prometheus release end-to-end (with different SD mechanisms) for a while and checks the results (evaluated expressions, alerts, etc.) for sanity.

Further we also lack proper benchmarks and this doc is also relevant: https://docs.google.com/document/d/16sCYq5EcB9_NFmnjtQ6WYmQ6tmt6CxYzsDL2ibrqPK8/edit

[krasi-georgiev]

really good idea!

[krasi-georgiev]

@gouthamve you can close this as I will track it in crosscloudci/crosscloudci#11

[sipian]

@krasi-georgiev
I am interested in this topic. Will like to discuss more about it