Remote write 2

[mmanciop]

Clean up the implementation of Prometheus Remote Write in Grafana Agent.

[sed-i]

Note that both prom libs have PRs in review.

[mmanciop]

For debugging purposes, we should annotate the Grafana Agent config with YAML comments how the IPs of the prometheus instances we remote-write to maps to their Juju identity:

integrations:
  prometheus_remote_write:
  - url: http://10.1.250.232:9090/api/v1/write  # Juju model X;Juju application Y; Juju unit Z
  - url: http://10.1.250.233:9090/api/v1/write # Juju model X;Juju application Y; Juju unit W
prometheus:
  configs:
  - name: agent_scraper
    remote_write:
    - url: http://10.1.250.232:9090/api/v1/write  # Comment same as previous case
    - url: http://10.1.250.233:9090/api/v1/write

[mmanciop]

In local testing, the Grafana Agent built from this PR is failing to push downstream to Prometheus the alert rules from the Cassandra charm.

[sed-i]

Re itest failing: I see similar errors too elsewhere.

[github-actions]

Libraries are not up to date with their remote counterparts. If this was
not intentional, run charmcraft fetch-libs and commit the updated libs
to your PR branch.

stdout

Library charms.loki_k8s.v0.loki_push_api was already up to date in version 0.10.                                                                               
Library charms.observability_libs.v0.kubernetes_service_patch was already up to date in version 0.6.                                                           
Library charms.prometheus_k8s.v0.prometheus_remote_write updated to version 0.4.                                                                               
Library charms.prometheus_k8s.v0.prometheus_scrape updated to version 0.18.    

stderr

[mmanciop]

Alert rule propagation from a prometheus_scrape relation over a prometheus_remote_write one does not seem to work in the following scenario:

juju add-model cos
juju deploy cos-lite --channel=edge --trust
juju offer prometheus:receive-remote-write prw
juju add-model agent
juju deploy ./*.charm grafana-agent --resource agent-image='grafana/agent:v0.20.1'
juju deploy spring-music --channel=edge
juju relate spring-music:metrics-endpoint grafana-agent
juju consume cos.prw prometheus
juju relate grafana-agent prometheus

Then open the Prometheus ui, go to the alert rules, despair: metrics from Spring Music come in, alert rules for Spring Music do not, while the alert rules for Grafana Agent itself do.

[dstathis]

Alert rule propagation does not seem to work in the following scenario:

1. juju add-model cos
2. juju deploy cos-lite --channel=edge --trust
3. juju offer prometheus:send-remote-write prw
4. juju add-model agent
5. juju deploy ./*.charm grafana-agent ...
6. juju deploy spring-music --channel=edge
7. juju relate spring-music:metrics-endpoint grafana-agent
8. juju consume cos.prw prometheus
9. juju relate grafana-agent prometheus
10. Open the Prometheus ui, go to the alert rules, despair

Metrics from Spring Music come in, alert rules do not

@mmanciop I tried this and it's working just fine for me. Could you possibly do it again and copy paste the exact commands?

[mmanciop]

Alert rule propagation does not seem to work in the following scenario:

1. juju add-model cos
2. juju deploy cos-lite --channel=edge --trust
3. juju offer prometheus:send-remote-write prw
4. juju add-model agent
5. juju deploy ./*.charm grafana-agent ...
6. juju deploy spring-music --channel=edge
7. juju relate spring-music:metrics-endpoint grafana-agent
8. juju consume cos.prw prometheus
9. juju relate grafana-agent prometheus
10. Open the Prometheus ui, go to the alert rules, despair

Metrics from Spring Music come in, alert rules do not

@mmanciop I tried this and it's working just fine for me. Could you possibly do it again and copy paste the exact commands?

I think I found the PBCAK. I was building the agent from the wrong commit.

[github-actions]

Libraries are not up to date with their remote counterparts. If this was
not intentional, run charmcraft fetch-libs and commit the updated libs
to your PR branch.

stdout

Library charms.loki_k8s.v0.loki_push_api was already up to date in version 0.10.                                                                               
Library charms.observability_libs.v0.kubernetes_service_patch was already up to date in version 0.6.                                                           
Library charms.prometheus_k8s.v0.prometheus_remote_write updated to version 0.4.                                                                               
Library charms.prometheus_k8s.v0.prometheus_scrape updated to version 0.18.    

stderr

[github-actions]

Libraries are not up to date with their remote counterparts. If this was
not intentional, run charmcraft fetch-libs and commit the updated libs
to your PR branch.

stdout

Library charms.loki_k8s.v0.loki_push_api was already up to date in version 0.10.                                                                               
Library charms.observability_libs.v0.kubernetes_service_patch was already up to date in version 0.6.                                                           
Library charms.prometheus_k8s.v0.prometheus_remote_write updated to version 0.4.                                                                               
Library charms.prometheus_k8s.v0.prometheus_scrape updated to version 0.18.    

stderr

[github-actions]

Libraries are not up to date with their remote counterparts. If this was
not intentional, run charmcraft fetch-libs and commit the updated libs
to your PR branch.

stdout

Library charms.loki_k8s.v0.loki_push_api was already up to date in version 0.10.                                                                               
Library charms.observability_libs.v0.kubernetes_service_patch was already up to date in version 0.6.                                                           
Library charms.prometheus_k8s.v0.prometheus_remote_write updated to version 0.4.                                                                               
Library charms.prometheus_k8s.v0.prometheus_scrape updated to version 0.18.    

stderr

[github-actions]

Libraries are not up to date with their remote counterparts. If this was
not intentional, run charmcraft fetch-libs and commit the updated libs
to your PR branch.

stdout

Library charms.loki_k8s.v0.loki_push_api was already up to date in version 0.10.                                                                               
Library charms.observability_libs.v0.kubernetes_service_patch was already up to date in version 0.6.                                                           
Library charms.prometheus_k8s.v0.prometheus_remote_write updated to version 0.4.                                                                               
Library charms.prometheus_k8s.v0.prometheus_scrape updated to version 0.18.    

stderr

[github-actions]

Libraries are not up to date with their remote counterparts. If this was
not intentional, run charmcraft fetch-libs and commit the updated libs
to your PR branch.

stdout

Library charms.loki_k8s.v0.loki_push_api was already up to date in version 0.10.                                                                               
Library charms.observability_libs.v0.kubernetes_service_patch was already up to date in version 0.6.                                                           
Library charms.prometheus_k8s.v0.prometheus_remote_write updated to version 0.4.                                                                               
Library charms.prometheus_k8s.v0.prometheus_scrape updated to version 0.18.    

stderr

[rbarry82]

Tests are solid, love seeing the mocking gone, rewrite rule is nice.

Feel free to just resolve any of my comments if you don't expect another push. They're not critical in any way.