Merge pull request #69423 from openshift-cherrypick-robot/cherry-pick…

…-68228-to-enterprise-4.15

[enterprise-4.15] OBSDOCS-479: Add flow control / rate limit docs

_topic_maps/_topic_map.yml

@@ -2599,6 +2599,11 @@ Topics:
     File: custom-logging-alerts
   - Name: Troubleshooting logging alerts
     File: troubleshooting-logging-alerts
+- Name: Performance and reliability tuning
+  Dir: performance_reliability
+  Topics:
+  - Name: Flow control mechanisms
+    File: logging-flow-control-mechanisms
 - Name: Troubleshooting Logging
   Dir: troubleshooting
   Distros: openshift-enterprise,openshift-origin

_topic_maps/_topic_map_osd.yml

@@ -1096,6 +1096,11 @@ Topics:
     File: custom-logging-alerts
   - Name: Troubleshooting logging alerts
     File: troubleshooting-logging-alerts
+- Name: Performance and reliability tuning
+  Dir: performance_reliability
+  Topics:
+  - Name: Flow control mechanisms
+    File: logging-flow-control-mechanisms
 - Name: Troubleshooting Logging
   Dir: troubleshooting
   Topics:

_topic_maps/_topic_map_rosa.yml

@@ -1305,6 +1305,11 @@ Topics:
     File: custom-logging-alerts
   - Name: Troubleshooting logging alerts
     File: troubleshooting-logging-alerts
+- Name: Performance and reliability tuning
+  Dir: performance_reliability
+  Topics:
+  - Name: Flow control mechanisms
+    File: logging-flow-control-mechanisms
 - Name: Troubleshooting Logging
   Dir: troubleshooting
   Topics:

logging/performance_reliability/_attributes

@@ -0,0 +1 @@
+../../_attributes/

logging/performance_reliability/images

@@ -0,0 +1 @@
+../../images/

logging/performance_reliability/logging-flow-control-mechanisms.adoc

@@ -0,0 +1,44 @@
+:_mod-docs-content-type: ASSEMBLY
+include::_attributes/common-attributes.adoc[]
+include::_attributes/attributes-openshift-dedicated.adoc[]
+[id="logging-flow-control-mechanisms"]
+= Flow control mechanisms
+:context: logging-flow-control-mechanisms
+
+toc::[]
+
+If logs are produced faster than they can be collected, it can be difficult to predict or control the volume of logs being sent to an output.
+Not being able to predict or control the volume of logs being sent to an output can result in logs being lost. If there is a system outage and log buffers are accumulated without user control, this can also cause long recovery times and high latency when the connection is restored.
+
+As an administrator, you can limit logging rates by configuring flow control mechanisms for your {logging}.
+
+[id="logging-configuring-flow-control-benefits"]
+== Benefits of flow control mechanisms
+
+* The cost and volume of logging can be predicted more accurately in advance.
+* Noisy containers cannot produce unbounded log traffic that drowns out other containers.
+* Ignoring low-value logs reduces the load on the logging infrastructure.
+* High-value logs can be preferred over low-value logs by assigning higher rate limits.
+
+[id="logging-configuring-flow-control-about-rate-limits"]
+== Configuring rate limits
+
+Rate limits are configured per collector, which means that the maximum rate of log collection is the number of collector instances multiplied by the rate limit.
+
+Because logs are collected from each node's file system, a collector is deployed on each cluster node. For example, in a 3-node cluster, with a maximum rate limit of 10 records per second per collector, the maximum rate of log collection is 30 records per second.
+
+Because the exact byte size of a record as written to an output can vary due to transformations, different encodings, or other factors, rate limits are set in number of records instead of bytes.
+
+You can configure rate limits in the `ClusterLogForwarder` custom resource (CR) in two ways:
+
+Output rate limit:: Limit the rate of outbound logs to selected outputs, for example, to match the network or storage capacity of an output. The output rate limit controls the aggregated per-output rate.
+
+Input rate limit:: Limit the per-container rate of log collection for selected containers.
+
+include::modules/logging-set-output-rate-limit.adoc[leveloffset=+1]
+
+[role="_additional-resources"]
+.Additional resources
+* xref:../../logging/log_collection_forwarding/logging-output-types.adoc#logging-output-types[Log output types]
+
+include::modules/logging-set-input-rate-limit.adoc[leveloffset=+1]

logging/performance_reliability/modules

@@ -0,0 +1 @@
+../../modules/

logging/performance_reliability/snippets

@@ -0,0 +1 @@
+../../snippets/

modules/logging-create-clf.adoc

@@ -19,7 +19,7 @@ You need administrator permissions for the namespace where you create the `Clust
 .ClusterLogForwarder resource example
 [source,yaml]
 ----
-apiVersion: "logging.openshift.io/v1"
+apiVersion: logging.openshift.io/v1
 kind: ClusterLogForwarder
 metadata:
   name: <log_forwarder_name> <1>

modules/logging-set-input-rate-limit.adoc

@@ -0,0 +1,76 @@
+// Module included in the following assemblies:
+//
+// * logging/performance_reliability/logging-flow-control-mechanisms.adoc
+
+:_mod-docs-content-type: PROCEDURE
+[id="logging-set-input-rate-limit_{context}"]
+= Configuring log forwarder input rate limits
+
+You can limit the rate of incoming logs that are collected by configuring the `ClusterLogForwarder` custom resource (CR). You can set input limits on a per-container or per-namespace basis.
+
+.Prerequisites
+
+* You have installed the {clo}.
+* You have administrator permissions.
+
+.Procedure
+
+. Add a `maxRecordsPerSecond` limit value to the `ClusterLogForwarder` CR for a specified input.
++
+The following examples show how to configure input rate limits for different scenarios:
++
+.Example `ClusterLogForwarder` CR that sets a per-container limit for containers with certain labels
+[source,yaml]
+----
+apiVersion: logging.openshift.io/v1
+kind: ClusterLogForwarder
+metadata:
+# ...
+spec:
+# ...
+  inputs:
+    - name: <input_name> <1>
+      application:
+        selector:
+          matchLabels: { example: label } <2>
+      limitPerContainer:
+          maxRecordsPerSecond: 0 <3>
+# ...
+----
+<1> The input name.
+<2> A list of labels. If these labels match labels that are applied to a pod, the per-container limit specified in the `maxRecordsPerSecond` field is applied to those containers.
+<3> Configures the rate limit. Setting the `maxRecordsPerSecond` field to `0` means that no logs are collected for the container. Setting the `maxRecordsPerSecond` field to some other value means that a maximum of that number of records per second are collected for the container.
++
+.Example `ClusterLogForwarder` CR that sets a per-container limit for containers in selected namespaces
+[source,yaml]
+----
+apiVersion: logging.openshift.io/v1
+kind: ClusterLogForwarder
+metadata:
+# ...
+spec:
+# ...
+  inputs:
+    - name: <input_name> <1>
+      application:
+        namespaces: [ example-ns-1, example-ns-2 ] <2>
+      limitPerContainer:
+        maxRecordsPerSecond: 10 <3>
+    - name: <input_name>
+      application:
+        namespaces: [ test ]
+      limitPerContainer:
+          maxRecordsPerSecond: 1000
+# ...
+----
+<1> The input name.
+<2> A list of namespaces. The per-container limit specified in the `maxRecordsPerSecond` field is applied to all containers in the namespaces listed.
+<3> Configures the rate limit. Setting the `maxRecordsPerSecond` field to `10` means that a maximum of 10 records per second are collected for each container in the namespaces listed.
+
+. Apply the `ClusterLogForwarder` CR:
++
+.Example command
+[source,terminal]
+----
+$ oc apply -f <filename>.yaml
+----

modules/logging-set-output-rate-limit.adoc

@@ -0,0 +1,48 @@
+// Module included in the following assemblies:
+//
+// * logging/performance_reliability/logging-flow-control-mechanisms.adoc
+
+:_mod-docs-content-type: PROCEDURE
+[id="logging-set-output-rate-limit_{context}"]
+= Configuring log forwarder output rate limits
+
+You can limit the rate of outbound logs to a specified output by configuring the `ClusterLogForwarder` custom resource (CR).
+
+.Prerequisites
+
+* You have installed the {clo}.
+* You have administrator permissions.
+
+.Procedure
+
+. Add a `maxRecordsPerSecond` limit value to the `ClusterLogForwarder` CR for a specified output.
++
+The following example shows how to configure a per collector output rate limit for a Kafka broker output named `kafka-example`:
++
+.Example `ClusterLogForwarder` CR
+[source,yaml]
+----
+apiVersion: logging.openshift.io/v1
+kind: ClusterLogForwarder
+metadata:
+# ...
+spec:
+# ...
+  outputs:
+    - name: kafka-example <1>
+      type: kafka <2>
+      limit:
+        maxRecordsPerSecond: 1000000 <3>
+# ...
+----
+<1> The output name.
+<2> The type of output.
+<3> The log output rate limit. This value sets the maximum link:https://kubernetes.io/docs/reference/kubernetes-api/common-definitions/quantity/[Quantity] of logs that can be sent to the Kafka broker per second. This value is not set by default. The default behavior is best effort, and records are dropped if the log forwarder cannot keep up. If this value is `0`, no logs are forwarded.
+
+. Apply the `ClusterLogForwarder` CR:
++
+.Example command
+[source,terminal]
+----
+$ oc apply -f <filename>.yaml
+----