Allow changes in remote write api endpoint secret

[QuentinBisson]

It is currently impossible to edit the remote write api endpoint secret because we made it immutable in the past because the remote write password value could not be changed in every reconciliation.

The manual workaround consists in deletint the secrets manually on all installations which is error-prone and cumbersome.

As the current instability issues we have with the agent will most likely make us need to tune our current configuration of the prometheus remote write, this PR ensure we can change the already existing values for each agent without the need of a manual intervention.

With this PR, we can now tune the agent remote write configuration while keeping the previously generated password, hence not causing any remote write errors due to a password mismatch between the one configured in the agent (that can take up to 5 minutes to be updated) and the one configured in the MC basic auth ingress (which is updated whenener PMO runs).

Please note thatthis PR will also regenerate the existing secret if it is immutable upon release. This means the aforementioned remote write errors will happen for 5-10 minutes after the release but will then not be an issue anymore.

Checklist

I have:

• Described why this change is being introduced
• Separated out refactoring/reformatting in a dedicated PR
• Updated changelog in CHANGELOG.md

[tityosbot]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[QuentinBisson]

Tested successfully on Gremlin, Just had some issues with marshalling and nil pointers :(

[QuantumEnigmaa]

Do those issues would prevent the changes induced by this PR to work on some installations ?

[QuentinBisson]

Nope that was me not knowing how to code

[marieroque]

Would it have been harder to split the current secret into an immutable secret and a configmap ?

[QuentinBisson]

harder, no, but the migration would have been harder and took longer. This allows us to update the config now, whereas moving to a configmap would require a new observability bundle release.

[marieroque]

harder, no, but the migration would have been harder and took longer. This allows us to update the config now, whereas moving to a configmap would require a new observability bundle release.

Ok I looked at the code. It's hard to see possible mistake!

Did you test side case like the one with a new cluster with no remote write secret ? Is the secret created correctly ?

[QuentinBisson]

Yes it's created fine, I tested on a working cluster and non working cluster

[marieroque]

Yes it's created fine, I tested on a working cluster and non working cluster

LGTM

[QuentinBisson]

For the migration to a CM and secret, we would basically have to first create a CM with same data as the secret, add it to the bundle, wait for it to be deployed everywhere, then remove the duplicate section from the secret. That will take a long time :(

[QuentinBisson]

We need to make this happen eventually but I do not have the bandwith and I think doing the promtail stuff will help us figure out a clean solution