bugfix: Remove whitespace and escape characters from URLs that are taken from secrets

[Jammicus]

When decoding from a kubernetes secret the slack url may have whitespace or escape characters appended to its suffix or prefix, this removes these so that it can correctly be parsed into alertmanager config

fixes #4037

Release Note Template (will be copied)

Fixed bug in Alertmanager config where URLS that are taken from Kubernetes secrets might contain whitespace or newline characters

[simonpasquier]

it would make sense for all URLs configured via secrets

[Jammicus]

Failing test for TestAllNS/y/PromPreserveUserAddedMetadata, will fix when doing the Pr comments

[simonpasquier]

PromPreserveUserAddedMetadata is flaky (I thought we had an issue about it but I can't find it right now). Feel free to investigate why the test fails sometimes but it's a bit of a rabbit hole :)

[Jammicus]

Haha I see what you mean about a rabbit hole....

Looks like its a timing issue when updating during reconciling the secrets annotations and labels, from what I can see happening is:

1. A series of kubernetes objects have custom annotations and labels added to it, with a secret being the last one
2. Reconciliation happens via scaling up replicas of the prometheus CRD
3. Sometimes the secret loses the custom annotations and labels, causing the test to fail

Adding a wait between adding the labels + annotations and reconciliation does not seem to help. Interestingly, moving the ordering of which resources have their labels/annotations changed first does affect it, so putting moving the secret from being the last to be modified to being second from last completely solves the issue. The object that is now last (an endpoint) always seems to keep its annotations and labels

[Jammicus]

Raised #4070 for the sporadic test failure

[Jammicus]

Updated to include other resources that support taking URLs from secrets

[paulfantom]

lgtm