feat: permit to run prometheus in agent mode

[nerzhul]

Description

Describe the big picture of your changes here to communicate to the maintainers why we should accept this pull request.
If it fixes a bug or resolves a feature request, be sure to link to that issue.

At Veepee we have very big prometheus in federation on our clusters. Some have more than 50GB memory and are pretty huge to federate.
Instead of federating we are considering to use remote_write feature and agent mode in order to publish the TSDB to the long term storage prometheus and thanos clusters.

If i enable agent feature we have prometheus crash because some configurations or flags are setuped. This include some TSDB retention flags, but also some generated configurations like rule_files.

This fix permit to run prometheus as a pure agent mode and disable the querier. on our dev cluster we reduce memory footprint by 60% on all our prometheus shards.

Type of change

What type of changes does your code introduce to the Prometheus operator? Put an x in the box that apply.

• CHANGE (fix or feature that would cause existing functionality to not work as expected)
• FEATURE (non-breaking change which adds functionality)
• BUGFIX (non-breaking change which fixes an issue)
• ENHANCEMENT (non-breaking change which improves existing functionality)
• NONE (if none of the other choices apply. Example, tooling, build system, CI, docs, etc.)

Changelog entry

Please put a one-line changelog entry below. This will be copied to the changelog file during the release process.

fix: permit prometheus to run in agent only mode

[simonpasquier]

👋 thanks for your PR! As you can read in #3989, the consensus among maintainers is that we want a dedicated PrometheusAgent CRD rather than piggy-backing on the existing Prometheus CRD. And at least 2 contributors have expressed interest in moving this forward.

Having said that it's already possible to use the Prometheus CRD to run in agent mode with a few tweaks (see #3989 (comment) for details). If this doesn't work for you, let us know.

[nerzhul]

hello @simonpasquier thanks for the interesting cheat on args, i will test it tomorrow, i was more close to code than overwriting the container myself as some args are generated by operator :)

[simonpasquier]

superseded by #5385