Add support for missing objectstorage secret for thanos store

[vprashar2929]

This PR:

• Adds support for missing object-storage secret which is required by Thanos store. The unavailability causes Thanos store pod to crash.
• Adds support to deploy minio as S3 for Thanos store
• Improve the way to deploy/teardown manifests locally
• Bump up Thanos image in manifests

[saswatamcode]

Seems like lint/build is failing. Could you take a look @vprashar2929?

[vprashar2929]

Done @saswatamcode Now everything is green 😄

[saswatamcode]

So taking a deeper look here, I think we should do things a bit differently.

Let's not define any minio resources within manifests but keep it to the examples/development-minio dir instead, so as to not block any downstream user simply copying manifests. Also importing observatorium is potentially a cyclic import here.

Perhaps we can define a minio.libsonnet jsonnet lib here, and generate the output of that in examples/development-minio?

[saswatamcode]

Hmm. Maybe we can define minio.libsonnet here?
Let's not create a cyclic import. Currently, observatorium/observatorium also relies on kube-thanos as a dependency.

[vprashar2929]

Sure defining minio.libsonnet here sounds good

[saswatamcode]

So I don't think having this within manifests is a good idea. Many downstream users simply want to copy over the manifests YAML and define their own S3 secrets and parameters (directly skipping jsonnet generation).

Including the minio secret here, would lead to an extra step for them.

[vprashar2929]

Yes I think moving the minio-related resources to examples/development-minio is the better way to proceed.

[saswatamcode]

LGTM!