examples: add example usage of prometheus-agent

[paulfantom]

Description

Describe the big picture of your changes here to communicate to the maintainers why we should accept this pull request.
If it fixes a bug or resolves a feature request, be sure to link to that issue.

PoC for running prometheus-agent with kube-prometheus.

NOTE: I do not have time to finalize this, so I welcome everyone wants to step up and finish this PR.

What needs to be improved:

• remote write configuration should be a first class config included in https://github.com/prometheus-operator/kube-prometheus/blob/main/jsonnet/kube-prometheus/components/prometheus.libsonnet#L1-L39
• alertmanagerName shouldn't be a required field. Lack of the field should remove alerting config section from prometheus object.

Type of change

What type of changes does your code introduce to the kube-prometheus? Put an x in the box that apply.

• CHANGE (fix or feature that would cause existing functionality to not work as expected)
• FEATURE (non-breaking change which adds functionality)
• BUGFIX (non-breaking change which fixes an issue)
• ENHANCEMENT (non-breaking change which improves existing functionality)
• NONE (if none of the other choices apply. Example, tooling, build system, CI, docs, etc.)

Changelog entry

Please put a one-line changelog entry below. Later this will be copied to the changelog file.

example of running prometheus-agent with kube-prometheus

[paulfantom]

Needs prometheus-operator/prometheus-operator#4417

[ArthurSens]

Do you want to enable us to work on your fork to get this one done? 🙂

[paulfantom]

The "Allow edits and access to secrets by maintainers" checkbox is checked, so you should've been able to work on this 🤔

[ArthurSens]

Hmmm that's awkward 😕

I've opened a pull request to your fork.

We still need some work left though, after running the agent in a cluster I've got this error:

ts=2021-11-29T16:05:15.753Z caller=main.go:195 level=info msg="Experimental agent mode enabled."
The following flag(s) can not be used in agent mode: ["--storage.tsdb.path" "--storage.tsdb.retention.time"]

[ArthurSens]

I've created another PR to your fork :)

[ArthurSens]

Yet another PR to your fork 😬

[paulfantom]

One last nit in form of removing prometheus version pinning (the whole project is already at 2.32+).

I think this is ready to go. @ArthurSens @philipgough could you take one more look and possibly merge this?

[ArthurSens]

All good from my side.