Alerting: Use configured headers for external alertmanager

[JohnnyQQQQ]

What is this feature?

We pass the configured headers from the alertmanager datasource to the external alertmanager call if any are configured.

Why do we need this feature?

Customers who are using an external alertmanager that need some headers, i.e. a multi tenant alertmanager from mimir or cortex.

Who is this feature for?

[Add information on what kind of user the feature is for.]

Which issue(s) does this PR fix?:

Fixes #60807

Special notes for your reviewer:

All changes were tested manually by me.

We did copy notifier.go from upstream and should try to upstream those changes afterward as they might be useful to a broader audience.

[yuri-tceretian]

I do not think copying whole client is a good idea. We have our fork of prometheus alertmanager and I think it would make more sense to update it and then use it in sender.

[JohnnyQQQQ]

I do not think copying whole client is a good idea. We have our fork of prometheus alertmanager and I think it would make more sense to update it and then use it in sender.

This code is from prometheus/prometheus and not from the prometheus/alertmanager.

Josh and me did an offline code review and agreed to move forward with this but try not to copy the config.go file. So I will submit some changes in the near future.

[yuri-tceretian]

Then I would appreciate more information about why the decision was to vendor file from upstream instead of updating our fork and then seek for an opportunity to upstream it.

I was pointed out that the copy was made from not alertmanager code but from prometheus.

[gotjosh]

Wow! Impressive job 👏👏👏👏👏👏👏

Very clean on the copy/paste from Prometheus - I hope we can upstream some of this work by making a good case around it.

Some parts of the router implementation can do with a bit of a cleanup; please see my comments.

[grafanabot]

This pull request was removed from the 9.5.0 milestone because 9.5.0 is currently being released.

[gotjosh]

LGTM

Beautiful 👏