*: Refactor to not mutate global objects

[brancz]

This patch introduces a few changes:

• Move all manifests to local configurations and non-merged local interactions. Which slightly changes the config paradigm as things have to be more declarative than modifying potentially existing "global" state.
• Makes the minio dev example work in any namespace
• Adds more declarative app labels defined by Kubernetes. This may cause users to have to remove and re-create manifests. We should carefully review this and see if we want any other label for this as we shouldn't change this again.
• Adds optional PodDisruptionBudget for Thanos receive.
• Moves ServiceMonitors to optional patches under each component .withServiceMonitor.

@metalmatze @squat @bwplotka @kakkoyun

Having reviewed this myself a few times, I recommend reading the generated manifests first, and then reviewing the new jsonnet files in their entity instead of comparing to the previous one.

[metalmatze]

Skimming over it on mobile made me very excited. I'll give it another look on a computer after FOSDEM. 👍

[brancz]

Sounds good. I think I just had a better idea for persistent volume claim templates anyways :)

[kakkoyun]

This looks great.

1. I think we should fix naming of the components, as it's discussed in Move components names to Ruler, Compactor, Querier (-er) thanos#1871.

receive should be receiver
query should be querier

2. You also removed sidecar. Which is ok, I think. We didn't have anything significant in that file. However, it would be good to document this. And maybe we can add a helper function to add sidecars to queriers. Just for convenience. This could be handled in another PR, I just wanted to note it.

[kakkoyun]

@brancz My only concern about naming is inconsistency. I'm totally fine with keeping configuration names in sync with commands. In that case, we should rename querier to query and compactor to compact.

My understanding from that issue (thanos-io/thanos#1871) was, we would eventually move to *tor naming schema, thus my comment.

And it's more clear now. Just to recap we will keep structural things as in commands and only rename documentation, alert message etc.

[brancz]

Looks like we have consensus in thanos-io/thanos#1871, I'll do the changes accordingly here.

[metalmatze]

Here's my the progress on moving my personal Thanos cluster to this refactored kube-thanos version.
I also put the diff to my thanos.yaml in there. A few things still missing but overall, this is an amazing improvement!

https://gist.github.com/metalmatze/7280c8101024d0edbccd5fbedb82a2c1

[metalmatze]

Let's do follow ups, if necessary, in new PRs! :)
Thaaaaaaaaanks soo much for this! 🎉