Add Vector & update Kubewatch configuration

[jb3]

This PR introduces Vector for log shipping amongst other QoL changes for our monitoring stack.

Vector is a very lightweight (~50mb) logging agent that can accept logs from a range of sources (similar to Alloy) and ship them to a range of sinks (in our case Loki).

We have switched Kubewatch from a custom deployment that sent data to Discord to a Helm deployment that will send structured JSON to Vector which will in turn push it into the Loki sink.

This allows us to keep track of events within our Kubernetes cluster over the long term in a more structured and useful fashion than just pumping information into a Discord channel.

Closes #347.

[jchristgit]

Just to make sure I understand this, the path for lines of text in our system is now:

(Process who is emitting logs) -> Vector -> Loki -> S3 server -> disk.

Is this correct?

[jb3]

Sort of.

I can add some documentation to our docs site as part of this to show it.

We have Loki as a final aggregation stage and sink, this will in turn write to S3 for long term storage.

Anything can write into Loki, in our case we have three things writing:

• Alloy as a daemonset (which ships Kubernetes logs)
• Alloy on our Ansible machines (which ships the system journal and tails any files we configure ourselves, like PostgreSQL logs)
• Vector as a single instance within Kubernetes to pick up bits we do not otherwise cover, by exposing things like a HTTP API that allows us to remap and transform data.

Vector does not replace Alloy, it's just a very lightweight translation layer for data which cannot otherwise run through Alloy.

In theory Vector supports the same features as Alloy so we could standardise on that as a solution across our nodes, but for now it sits well as a tiny container that just exposes a good-old HTTP API.

[ChrisLovering]

lgtm

[jb3]

/merge

[pydis-ff-bot]

✅ Successfully fast-forwarded commits from python-discord:jb3/vector onto main

[github-actions]

Thanks!

[jchristgit]

Thanks. I am glad to announce that in the year of 2026, I no longer comprehend the """state of the art""" in logging stacks. Hopefully I am no longer on this planet by the time systemd-journald is implemented as a microservice.