(cos-agent) Use peer relation data for communicating all principals' data

[sed-i]

Issue

Subordinate charms only see one unit over the subordinated relation - the principal unit. I.e. subordinate units cannot iterate over relation data from other units.
Combined with a leader guard, this means only one principal unit can get its files across - the one that is related to the subordinate leader.

Solution

• Provider: copy data from principal to peer unit data
• Requirer: refactor _fetch_data_from_relation
• Manual e2e test:
  • CMR: prom (metrics, alerts)
  • CMR: loki (logs, alerts)
  • CMR: grafana (dashboards)
• Unit tests

Fixes #158.

Context

Testing Instructions

1. Check out Add o11y integration zookeeper-operator#56 and copy over the cos-agent lib from this PR.
2. Pack zookeeper.
3. Deploy first-zk (two units), second-zk (one unit is enough) and gagent from this PR.
4. Relate both Zks to gagent over cos-agent.
5. juju show-unit ...
6. Good luck :)

Eventually we'd like to test something like this:

graph LR

subgraph user-model in a lxd controller

subgraph first-zk
first-zk/0
first-zk/1
end

subgraph second-zk
second-zk/0
end

subgraph agent
agent/0
agent/1
agent/2
end

first-zk ---|cos-agent| agent
first-zk/0 -.-|same VM| agent/0

second-zk/0 -.-|"same VM"| agent/2
second-zk ---|"  cos-agent &nbsp "| agent


end

agent ---|CMR - send-remote-write| prom
agent ---|CMR - send-remote-write| prom2

subgraph cos-model in a K8s controller
subgraph prom
prom/0
prom/1
end

subgraph prom2
prom2/0
end

end

Loading

Release Notes

Use peer relation data for communicating all principals' data.

[sed-i]

Hey @PietroPasotti
I refactored this a bit after realizing that we need to forward only some of the data from the principal to the subordinate leader.

graph LR

zk/0 --->|"
    metrics_alert_rules
    log_alert_rules
    dashboards
    metrics_scrape_jobs
    log_slots
"| gagent/i --->|"
    metrics_alert_rules
    log_alert_rules
    dashboards
"| gagent/leader* --->|"
    metrics_alert_rules
    log_alert_rules
    dashboards
"| cos

Loading

• I introduced two dataclasses. The main issue with them is that runtime json parsing errors give zero insight into the problem. I hope pydantic is capable informing why parsing a json string into a dataclass (model) failed.
• I was incrementally fixing bugs as moved along manual testing. Bug fixing isn't done but hopefully good enough for you to pick up.

[simskij]

Manually tested at 1c7d99ad10e344ae187f55f6473df5e71c2554a4, which worked great.