Long term storage solution

[etsauer]

By default prometheus only stores 2 weeks worth of data. In order for MDT tooling to be valuable, we need to store at least 6 months to a year's worth of History. We need to do some research into a long term data store for the stack.

[etsauer]

Here's one blog post that suggests a solution, but there may be more.

https://blog.openshift.com/thanos-long-term-storage-for-your-prometheus-metrics-on-openshift/

Also, looking at the Prometheus Storage docs, there's nothing that explicitly says that it should only be used for short-term storage, but it does point out that promehteus itself doesn't do much in the way of operationalizing its storage, and references some thrid-party storage integrations.

https://prometheus.io/docs/prometheus/latest/storage/

[ramius345]

I will start taking a look at this tomorrow. Thanks.

[garethahealy]

Theres already been some thanos work in the CoP:

• #172 Draft article on monitoring multiple prometheus clusters redhat-cop/uncontained.io#190

CC @robbbbh @AndreaFerraresi

[ramius345]

I'll have a look, thanks!

[ramius345]

I have a working solution on this using nooba for Openshift 4. However, when I try this on the CI CASL cluster, nooba is having issues installing with PV provisioning. Using version 2.0.9 of noobaa, I am able to create a backing store, but I am not able to get noobaa to finish its installation, it gets stuck waiting for something. Looking at the logs, its looking for AWS and Ceph provisioners.

We should be able to use the rest of the solution with any s3 compatible bucket provider. It might be time to start looking at Ceph or to just suggest straight up AWS s3.

[bbeaudoin]

Nooba aggregates S3 API providers, Ceph API and S3 are the default backing types, this cannot be block so if CI CASL doesn't support S3 API compatible storage you'll need something other than Nooba. Thanos is a good fit but the storage needs to be worked out.

[etsauer]

@bbeaudoin ok, I misunderstood what Noobaa was then. @ramius345 I think the default option for this project should be to use MinIO instead.

[sbailleu]

Functional code ready (iteration 1), code needs to be merged. Documentation to be completed.

[etsauer]

@tolarewaju3 now that @ramius345 work has been merged, here's what I think it left TODO:

• Come up with an opinionated installation method for MinIO that we can document, that at a minimum includes a TLS config
• Refactor the helm chart to allow the user to configure any Object Store provider for thanos
• Update documentation

[etsauer]

Oh, and one more:

• Repoint grafana to the thanos instance instead of directly to prometheus

[tolarewaju3]

Going to write documentation recommending minio helm chart. The current operator install doesn't work out of the box because of these two issues:

1. rbac issue with latest minio/operator#91
2. minioinstances.miniocontroller.min.io vs. minioinstances.miniooperator.min.io minio/operator#94

We can update the guide in the future when those are in master

[tolarewaju3]

Test PR is up. Still needs work on documentation & this task:

• Refactor the helm chart to allow the user to configure any Object Store provider for thanos

[themoosman]

Here are my testing steps:

1. Deployed the stack per the directions into a single namespace pelorus.
2. Deployed the exporters and loaded some data into Prometheus.
3. Verified the data appeared in the dashboard and via PromQL query
4. Delete the 2 Prometheus and checked for data. Neither the dashboard or a PromQL query returned any results.
5. Created a route for the MinIO instance to verify data was being written to MinIO. Minio shows no data in the buckets.

It appears that no data is being written to MinIO.

[tolarewaju3]

@themoosman Hmm. Ok. I will re-test this when I get some time. When I ran it, it took a bit to populate in minio. But it sounds like you already saw data in pelorus