feat!: add sumologic.events.enabled property (#2407)

This builds on event collection with OpenTelemetry Collector introduced in #2379, but changes how event collection is configured. It does so by removing the property `otelevents.enabled` and instead adding `sumologic.events.enabled` and `sumologic.events.provider`.

Also adding some more documentation around this and fixing some typos or misleading docs.

With this change, every property does exactly one thing:

- `sumologic.events.enabled` enables or disables events collection
- `sumologic.events.provider` switches between Fluentd and Otelcol

We also still support the (now deprecated) `fluentd.events.enabled` property to disable events collection with Fluentd.

CHANGELOG.md

@@ -14,7 +14,7 @@ This change affects you only if you have enabled traces with `sumologic.traces.e
 ### Added
 
 - feat(metrics): add service metrics [#2367]
-- feat(events): add experimental OT event collection [#2379]
+- feat(events): add experimental OT event collection [#2379], [#2407]
 
 ### Changed
 
@@ -41,6 +41,7 @@ This change affects you only if you have enabled traces with `sumologic.traces.e
 [#2390]: https://github.com/SumoLogic/sumologic-kubernetes-collection/pull/2390
 [#2379]: https://github.com/SumoLogic/sumologic-kubernetes-collection/pull/2379
 [#2405]: https://github.com/SumoLogic/sumologic-kubernetes-collection/pull/2405
+[#2407]: https://github.com/SumoLogic/sumologic-kubernetes-collection/pull/2407
 [Unreleased]: https://github.com/SumoLogic/sumologic-kubernetes-collection/compare/v2.10.0...main
 
 ## [v2.10.0]

deploy/README.md

@@ -28,6 +28,7 @@ Documentation links:
   - [Authenticating with container registry](./docs/Working_with_container_registries.md#authenticating-with-container-registry)
     - [Using pull secrets with `sumologic-kubernetes-collection` helm chart](./docs/Working_with_container_registries.md#authenticating-with-container-registry)
   - [Container log parsing (Docker, CRI-O, containerd)](./docs/ContainerLogs.md)
+  - [Collecting Kubernetes events](./docs/collecting-kubernetes-events.md)
   - Open Telemetry `beta`
     - [Open Telemetry with `sumologic-kubernetes-collection`](./docs/opentelemetry_collector.md)
     - [Comparison of Fluentd and Opentelemetry Collector functionality](./docs/fluentd_otc_comparison.md)

deploy/docs/collecting-kubernetes-events.md

@@ -0,0 +1,86 @@
+# Collecting Kubernetes events
+
+You can collect Kubernetes events from the Kubernetes API server and send them to Sumo Logic as logs.
+
+This feature is enabled by default.
+To disable it, set the `sumologic.events.enabled` property to `false`.
+
+Event collection is performed by the provider specified in `sumologic.events.provider`. This can be either `fluentd` for Fluentd (which is currently the default) or `otelcol` for OpenTelemetry Collector (which will be the default in a future release).
+You can switch the provider by setting the property:
+
+```yaml
+sumologic:
+  events:
+    provider: otelcol
+```
+
+Both providers work in the same way: they request all Kubernetes events from the Kubernetes API server.
+Note that the resource API used is [core v1][event_v1_core] and not [events.k8s.io/v1][event_events_k8s_io].
+The events are sent as logs in their original JSON format to Sumo Logic.
+
+Example Kubernetes event:
+
+```json
+{
+  "object": {
+    "apiVersion": "v1",
+    "count": 19736,
+    "eventTime": null,
+    "firstTimestamp": "2022-03-12T20:48:26Z",
+    "involvedObject": {
+      "apiVersion": "v1",
+      "fieldPath": "spec.containers{aws-node}",
+      "kind": "Pod",
+      "name": "aws-node-sshmk",
+      "namespace": "kube-system",
+      "resourceVersion": "55028103",
+      "uid": "96a623cd-e201-4ba5-9595-231cdf3da63d"
+    },
+    "kind": "Event",
+    "lastTimestamp": "2022-07-05T01:47:09Z",
+    "message": "Pulling image \"602401143452.dkr.ecr.us-west-1.amazonaws.com/amazon-k8s-cni:v1.7.5-eksbuild.1\"",
+    "metadata": {
+      "creationTimestamp": "2022-07-05T01:47:09Z",
+      "name": "aws-node-sshmk.16dbbd30f2200271",
+      "namespace": "kube-system",
+      "resourceVersion": "86640936",
+      "selfLink": "/api/v1/namespaces/kube-system/events/aws-node-sshmk.16dbbd30f2200271",
+      "uid": "d7a69a2e-3842-4f26-abec-f9949158f189"
+    },
+    "reason": "Pulling",
+    "reportingComponent": "",
+    "reportingInstance": "",
+    "source": {
+      "component": "kubelet",
+      "host": "ip-172-16-8-171.us-west-1.compute.internal"
+    },
+    "type": "Normal"
+  },
+  "timestamp": 1656985629543,
+  "type": "ADDED"
+}
+```
+
+## Configuration
+
+To configure event collection, see the following sections of the [values.yaml][values_yaml] file, depending on the provider used:
+
+- `fluentd.events` for Fluentd provider (the default)
+- `otelevents` for OpenTelemetry Collector provider
+
+Also see [OpenTelemetry Collector document][otelcol_config] for more details on configuring the Otelcol provider.
+
+## Disabling Kubernetes events collection
+
+To disable the collection of Kuebrnetes events, set the `sumologic.events.enabled` property to `false`:
+
+```yaml
+sumologic:
+  events:
+    enabled: false
+```
+
+[event_v1_core]: https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#event-v1-core
+[event_events_k8s_io]: https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#event-v1-events-k8s-io
+[values_yaml]: ../helm/sumologic/values.yaml
+[otelcol_config]: ./opentelemetry_collector.md#kubernetes-events

deploy/docs/opentelemetry_collector.md

@@ -12,9 +12,15 @@ We offer it as drop-in replacement for Fluentd in our collection.
 - [Scraping Containers Logs](#scraping-containers-logs)
 - [Persistence](#persistence)
   - [Enabling persistence](#enabling-persistence)
+    - [Enabling Opentelemetry Collector persistence by recreating StatefulSet](#enabling-opentelemetry-collector-persistence-by-recreating-statefulset)
+    - [Enabling Opentelemetry Collector persistence by creating temporary instances and removing earlier created](#enabling-opentelemetry-collector-persistence-by-creating-temporary-instances-and-removing-earlier-created)
   - [Disabling persistence](#disabling-persistence)
+    - [Disabling Opentelemetry Collector persistence by recreating StatefulSet](#disabling-opentelemetry-collector-persistence-by-recreating-statefulset)
+    - [Disabling Opentelemetry Collector persistence by creating temporary instances nd removing earlier created](#disabling-opentelemetry-collector-persistence-by-creating-temporary-instances-nd-removing-earlier-created)
 - [Traces](#traces)
   - [Load balancing using the gateway](#load-balancing-using-the-gateway)
+- [Kubernetes Events](#kubernetes-events)
+  - [Customizing OpenTelemetry Collector configuration](#customizing-opentelemetry-collector-configuration)
 
 ## Metrics
 
@@ -597,29 +603,32 @@ Refs:
 
 ## Kubernetes Events
 
-Opentelemetry Collector can be used to collect and enrich Kubernetes Events in lieu of FluentD.
-This is a drop-in replacement, and currently the OT configuration directly uses some of the FluentD configuration values.
-It can be enabled by setting:
+OpenTelemetry Collector can be used to collect and enrich Kubernetes events instead of Fluentd.
+This is a drop-in replacement. To do this, set the `sumologic.events.provider` to `otelcol`:
 
 ```yaml
 sumologic:
-  fluentd:
-    events:
-      enabled: false
-
-  otelevents:
-    enabled: true
+  events:
+    provider: otelcol
 ```
 
-For configurations which don't modify `sumologic.fluentd.events.overrideOutputConf`, this should be enough. See the configuration
-options under `sumologic.otelevents` in [values.yaml](../helm/sumologic/values.yaml).
+Currently the OT configuration directly uses the following Fluentd configuration values:
+
+- `fluentd.events.sourceCategory` to specify the [source category][source_category] for the event logs
+- `fluentd.events.sourceName` to specify the [source name][source_name] for the event logs
+
+For configurations that don't modify `sumologic.fluentd.events.overrideOutputConf`, this should be enough. See the configuration
+options under `otelevents` in [values.yaml](../helm/sumologic/values.yaml).
+
+[source_category]: https://help.sumologic.com/03Send-Data/Sources/04Reference-Information-for-Sources/Metadata-Naming-Conventions#Source_Categories
+[source_name]: https://help.sumologic.com/03Send-Data/Sources/04Reference-Information-for-Sources/Metadata-Naming-Conventions#Source_Name
 
 ### Customizing OpenTelemetry Collector configuration
 
-If the configuration options present under the `sumologic.otelevents` key aren't sufficient for your needs, you can override
+If the configuration options present under the `otelevents` key aren't sufficient for your needs, you can override
 the OT configuration directly. Be aware that doing this isn't subject to normal backwards-compatibility guarantees offered by
 this chart, so you'll need to be more careful during upgrades. The exact OT configuration emitted by the chart may change
 even in minor releases.
 
-In order to override the configuration, use the `sumologic.otelevents.config.override` key. This key takes a yaml object, whose
+In order to override the configuration, use the `otelevents.config.override` key. This key takes a yaml object, whose
 value is merged with the configuration generated by the Chart.