feat: add experimental OT event collection

[swiatekm]

Description

This adds event collection using OT. As we don't have an events subsection in the sumologic section, I can't implement a simple switch like we have for metadata, and instead need to add a new section and depend on the user not to enable both at once.

One thing of note is the approach to configuring OT. Instead of simply embedding the configuration in values.yaml, we instead treat it as a normal template, and then allow overriding it via a configuration field whose function will be similar to the current approach, but which is intended as a last resort escape hatch rather than the default way of configuring OT.

TODO:

• documentation
• comments for the more interesting features

---

Checklist

• Changelog updated

Testing performed

• Redeploy fluentd and fluentd-events pods
• Confirm events, logs, and metrics are coming in

[swiatekm]

Note: this currently includes #2388, I'll rebase once that's merged to make the config table changes clearer.

[github-actions]

This pull request contains invalid labels. Please remove all of the following labels: ['do-not-merge/hold']

[swiatekm]

This depends on SumoLogic/sumologic-otel-collector#620, and therefore probably on the 0.54.0 OT distro release.

[sumo-drosiek]

I would insist to test all possible values for statefulset (including sumologic key and so on)

[sumo-drosiek]

We can consider provider approach, to avoid both otelevents and fluentd being run at the same time

[swiatekm]

We can consider provider approach, to avoid both otelevents and fluentd being run at the same time

That's a breaking change though, as events don't have a section under sumologic. We can switch to that in 3.0.

[sumo-drosiek]

it is not until fluentd will be default choice

[swiatekm]

it is not until fluentd will be default choice

It is. If someone disables events by setting fluentd.events.enabled to false, and I add sumologic.events.enabled defaulting to true, it's either enabled, in which case it's a breaking change, or not, in which case we have a situation very similar to right now.

[sumo-drosiek]

I was thinking rather about flag sumologic.events.provider

[swiatekm]

I was thinking rather about flag sumologic.events.provider

Does that actually help? We run into problems if someone enables both FluentD and OT events. In that case, sumologic.events.provider could be a tiebreaker, but it's confusing that you need to set this and separately enable the thing you want.

[sumo-drosiek]

we could enable otc events by default and control it by sumologic.events.provider only, especially if we plan to have this flag in the future. If not, we can skip it