Loki alerts are not being pushed to AlertManager

[michaeldmitry]

Bug Description

When loki alerts are triggered, loki-k8s fails to push them to alert manager, so alerts are not being displayed.
By inspecting loki logs, the issue appears to be related to the alert manager api version used by loki.

To Reproduce

1. juju deploy cos-lite --trust
2. write a loki alert rule that you expect to trigger an alert frequently in "your_charm"
3. juju integrate "your_charm":logging loki:logging
4. wait for the alert to trigger
5. open alert manager dashboard (get from juju run traefik/0 show-proxied-endpoints)
6. The alert is not displayed
7. check logs of loki (kubectl logs -n <model_name> loki-0 -c loki)
8. search for "410 Gone"

Environment

juju version: 3.4.2
microk8s version: MicroK8s v1.28.7 revision 6532
loki version: latest/stable revision 128

Relevant log output

Additional context

No response