Send OVN logs to events API and Loki

[monstermunchkin]

This adds an OVN log monitor which listens for log messages on a unix socket. If messages are received, they are sent to the events API which will then forward them to Loki (if configured). The log monitor can be enabled by setting core.syslog_socket to true. The logs will only be sent to Loki, if loki.types contains ovn.

In this case, the ovn events can also be monitored using lxc monitor --type=ovn.

Setting up OVN

On Ubuntu, OVN uses the /etc/default/ovn-host environment file. To enable the OVN controller to send their logs via unix socket, one needs to add the following:

OVN_CTL_OPTS="--ovn-controller-log='-vsyslog:info --syslog-method=unix:/var/lib/lxd/syslog.socket'"

There are also --ovn-sb-log and --ovn-nb-log which take the same arguments. These need to be set in /etc/default/ovn-central though.

Specification https://discourse.ubuntu.com/t/send-ovn-logs-to-the-events-api/38445

[simondeziel]

LGTM but I'm curious to hear your opinion on the seek approach.

[tomponline]

We should not re-read the whole log file(s) each time they change.

[tomponline]

@monstermunchkin when you're ready for another review please can you click the refresh review button on my name top right so that I get a notification and it'll appear on my todo list on https://github.com/pulls/review-requested

Thanks

[simondeziel]

@tomponline @monstermunchkin ovn supports streaming logs to a Unix socket using syslog protocol. Would it be simpler to stream those through LXD then to Loki? That would avoid writing them to disk to read them back.

[tomponline]

@tomponline @monstermunchkin ovn supports streaming logs to a Unix socket using syslog protocol. Would it be simpler to stream those through LXD then to Loki? That would avoid writing them to disk to read them back.

That would be preferable by the sounds of it, do you have more details on how this works in a cluster?
Does each host send its own local logs to the local unix socket?

[simondeziel]

That's a good question re cluster, one I cannot answer. All I saw is https://www.man7.org/linux/man-pages/man8/ovn-controller.8.html which also mentions log levels can changed dynamically using ovs-appctl and that also includes the off level.

[monstermunchkin]

I've been playing around with the unix socket thing but didn't manage to get it working. I wasn't receiving any log messages for some reason.

However, what I did get working is OVN sending logs via UDP using --syslog-method=udp:localhost:<port>.

[tomponline]

Sounds promising. Although this approach would require more setup in ovn, it would avoid needing special microovn handling, and we could get microcloud to make the necessary configuration changes in both microovn and lxd during setup.

[tomponline]

cc @masnax @markylaing

[tomponline]

This is also missing the tests we discussed adding in our 1:1 wrt to using nc to send messages to the syslog listener and checking they appear in the lxc monitor.

[tomponline]

@monstermunchkin please let me know when tests pass and ill re-review. Thanks

[ru-fu]

Some fixes.

[monstermunchkin]

@tomponline all tests have passed.

[tomponline]

LGTM thanks