diff --git a/logging/log_collection_forwarding/cluster-logging-collector.adoc b/logging/log_collection_forwarding/cluster-logging-collector.adoc index 3f187dcc407d..0cc727bc232d 100644 --- a/logging/log_collection_forwarding/cluster-logging-collector.adoc +++ b/logging/log_collection_forwarding/cluster-logging-collector.adoc @@ -14,10 +14,6 @@ include::modules/configuring-logging-collector.adoc[leveloffset=+1] include::modules/creating-logfilesmetricexporter.adoc[leveloffset=+1] -include::modules/log-collector-resources-scheduling.adoc[leveloffset=+1] - -include::modules/cluster-logging-collector-pod-location.adoc[leveloffset=+1] - include::modules/cluster-logging-collector-limits.adoc[leveloffset=+1] [id="cluster-logging-collector-input-receivers"] diff --git a/logging/log_storage/cluster-logging-loki.adoc b/logging/log_storage/cluster-logging-loki.adoc index bc6e504eda57..1edf5bfbce09 100644 --- a/logging/log_storage/cluster-logging-loki.adoc +++ b/logging/log_storage/cluster-logging-loki.adoc @@ -18,8 +18,6 @@ include::modules/logging-loki-restart-hardening.adoc[leveloffset=+1] include::modules/logging-loki-reliability-hardening.adoc[leveloffset=+1] -include::modules/logging-loki-pod-placement.adoc[leveloffset=+1] - [role="_additional-resources"] .Additional resources * link:https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#podantiaffinity-v1-core[`PodAntiAffinity` v1 core Kubernetes documentation] @@ -63,4 +61,4 @@ include::modules/logging-loki-memberlist-ip.adoc[leveloffset=+1] * link:https://loki-operator.dev/docs/howto_connect_grafana.md/[Grafana Dashboard documentation] * link:https://loki-operator.dev/docs/object_storage.md/[Loki Object Storage documentation] * link:https://loki-operator.dev/docs/api.md/#loki-grafana-com-v1-IngestionLimitSpec[{loki-op} `IngestionLimitSpec` documentation] -* link:https://grafana.com/docs/loki/latest/operations/storage/schema/#changing-the-schema[Loki Storage Schema documentation] \ No newline at end of file +* link:https://grafana.com/docs/loki/latest/operations/storage/schema/#changing-the-schema[Loki Storage Schema documentation] diff --git a/logging/scheduling_resources/logging-node-selectors.adoc b/logging/scheduling_resources/logging-node-selectors.adoc index e39331d54223..e02f345267e7 100644 --- a/logging/scheduling_resources/logging-node-selectors.adoc +++ b/logging/scheduling_resources/logging-node-selectors.adoc @@ -10,7 +10,12 @@ toc::[] include::snippets/about-node-selectors.adoc[] include::modules/nodes-scheduler-node-selectors-about.adoc[leveloffset=+1] -include::modules/infrastructure-moving-logging.adoc[leveloffset=+1] + +include::modules/logging-loki-pod-placement.adoc[leveloffset=+1] + +include::modules/log-collector-resources-scheduling.adoc[leveloffset=+1] + +include::modules/cluster-logging-collector-pod-location.adoc[leveloffset=+1] [role="_additional-resources"] [id="additional-resources_logging-node-selection"] diff --git a/logging/scheduling_resources/logging-taints-tolerations.adoc b/logging/scheduling_resources/logging-taints-tolerations.adoc index 221f7d950390..86744f087dd9 100644 --- a/logging/scheduling_resources/logging-taints-tolerations.adoc +++ b/logging/scheduling_resources/logging-taints-tolerations.adoc @@ -10,10 +10,15 @@ toc::[] Taints and tolerations allow the node to control which pods should (or should not) be scheduled on them. include::modules/nodes-scheduler-taints-tolerations-about.adoc[leveloffset=+1] -include::modules/cluster-logging-logstore-tolerations.adoc[leveloffset=+1] -include::modules/cluster-logging-kibana-tolerations.adoc[leveloffset=+1] + +include::modules/logging-loki-pod-placement.adoc[leveloffset=+1] + include::modules/cluster-logging-collector-tolerations.adoc[leveloffset=+1] +include::modules/log-collector-resources-scheduling.adoc[leveloffset=+1] + +include::modules/cluster-logging-collector-pod-location.adoc[leveloffset=+1] + [role="_additional-resources"] [id="additional-resources_cluster-logging-tolerations"] == Additional resources diff --git a/machine_management/creating-infrastructure-machinesets.adoc b/machine_management/creating-infrastructure-machinesets.adoc index 71b3ffdfe4c1..546a2afa8cb3 100644 --- a/machine_management/creating-infrastructure-machinesets.adoc +++ b/machine_management/creating-infrastructure-machinesets.adoc @@ -8,10 +8,9 @@ toc::[] include::modules/machine-user-provisioned-limitations.adoc[leveloffset=+1] - You can use infrastructure machine sets to create machines that host only infrastructure components, such as the default router, the integrated container image registry, and the components for cluster metrics and monitoring. These infrastructure machines are not counted toward the total number of subscriptions that are required to run the environment. -In a production deployment, it is recommended that you deploy at least three machine sets to hold infrastructure components. Both OpenShift Logging and {SMProductName} deploy Elasticsearch, which requires three instances to be installed on different nodes. Each of these nodes can be deployed to different availability zones for high availability. This configuration requires three different machine sets, one for each availability zone. In global Azure regions that do not have multiple availability zones, you can use availability sets to ensure high availability. +In a production deployment, it is recommended that you deploy at least three machine sets to hold infrastructure components. {SMProductName} deploys Elasticsearch, which requires three instances to be installed on different nodes. Each of these nodes can be deployed to different availability zones for high availability. This configuration requires three different machine sets, one for each availability zone. In global Azure regions that do not have multiple availability zones, you can use availability sets to ensure high availability. include::modules/infrastructure-components.adoc[leveloffset=+1] @@ -22,7 +21,7 @@ To create an infrastructure node, you can xref:../machine_management/creating-in [id="creating-infrastructure-machinesets-production"] == Creating infrastructure machine sets for production environments -In a production deployment, it is recommended that you deploy at least three compute machine sets to hold infrastructure components. Both OpenShift Logging and {SMProductName} deploy Elasticsearch, which requires three instances to be installed on different nodes. Each of these nodes can be deployed to different availability zones for high availability. A configuration like this requires three different compute machine sets, one for each availability zone. In global Azure regions that do not have multiple availability zones, you can use availability sets to ensure high availability. +In a production deployment, it is recommended that you deploy at least three compute machine sets to hold infrastructure components. {SMProductName} deploys Elasticsearch, which requires three instances to be installed on different nodes. Each of these nodes can be deployed to different availability zones for high availability. A configuration like this requires three different compute machine sets, one for each availability zone. In global Azure regions that do not have multiple availability zones, you can use availability sets to ensure high availability. [id="creating-infrastructure-machinesets-clouds"] === Creating infrastructure machine sets for different clouds @@ -131,9 +130,8 @@ include::modules/infrastructure-moving-registry.adoc[leveloffset=+2] include::modules/infrastructure-moving-monitoring.adoc[leveloffset=+2] -include::modules/infrastructure-moving-logging.adoc[leveloffset=+2] - [role="_additional-resources"] .Additional resources - -* See xref:../monitoring/configuring-the-monitoring-stack.adoc#moving-monitoring-components-to-different-nodes_configuring-the-monitoring-stack[the monitoring documentation] for the general instructions on moving {product-title} components. +* xref:../monitoring/configuring-the-monitoring-stack.adoc#moving-monitoring-components-to-different-nodes_configuring-the-monitoring-stack[Moving monitoring components to different nodes] +* xref:../logging/scheduling_resources/logging-node-selectors.adoc#logging-node-selectors[Using node selectors to move logging resources] +* xref:../logging/scheduling_resources/logging-taints-tolerations.adoc#cluster-logging-logstore-tolerations_logging-taints-tolerations[Using taints and tolerations to control logging pod placement] diff --git a/modules/cluster-logging-kibana-tolerations.adoc b/modules/cluster-logging-kibana-tolerations.adoc deleted file mode 100644 index a3bdb7f0f9bd..000000000000 --- a/modules/cluster-logging-kibana-tolerations.adoc +++ /dev/null @@ -1,64 +0,0 @@ -// Module included in the following assemblies: -// -// * logging/scheduling_resources/logging-taints-tolerations.adoc - -:_mod-docs-content-type: PROCEDURE -[id="cluster-logging-kibana-tolerations_{context}"] -= Using tolerations to control the log visualizer pod placement - -You can use a specific key/value pair that is not on other pods to ensure that only the Kibana pod can run on the specified node. - -.Prerequisites - -* You have installed the {clo}, the {es-op}, and the {oc-first}. - -.Procedure - -. Add a taint to a node where you want to schedule the log visualizer pod by running the following command: -+ -[source,terminal] ----- -$ oc adm taint nodes =: ----- -+ -.Example command -[source,terminal] ----- -$ oc adm taint nodes node1 kibana=node:NoExecute ----- -+ -This example places a taint on `node1` that has key `kibana`, value `node`, and taint effect `NoExecute`. You must use the `NoExecute` taint effect. `NoExecute` schedules only pods that match the taint and remove existing pods that do not match. - -. Edit the `visualization` section of the `ClusterLogging` CR to configure a toleration for the Kibana pod: -+ -[source,yaml] ----- -apiVersion: logging.openshift.io/v1 -kind: ClusterLogging -metadata: -# ... -spec: -# ... - visualization: - type: kibana - kibana: - tolerations: - - key: kibana <1> - operator: Exists <2> - effect: NoExecute <3> - tolerationSeconds: 6000 <4> - resources: - limits: - memory: 2Gi - requests: - cpu: 100m - memory: 1Gi - replicas: 1 -# ... ----- -<1> Specify the key that you added to the node. -<2> Specify the `Exists` operator to require the `key`, value, and `effect` parameters to match. -<3> Specify the `NoExecute` effect. -<4> Optionally, specify the `tolerationSeconds` parameter to set how long a pod can remain bound to a node before being evicted. - -This toleration matches the taint created by the `oc adm taint` command. A pod with this toleration would be able to schedule onto `node1`. diff --git a/modules/infrastructure-moving-logging.adoc b/modules/infrastructure-moving-logging.adoc deleted file mode 100644 index a24e38ff13f3..000000000000 --- a/modules/infrastructure-moving-logging.adoc +++ /dev/null @@ -1,223 +0,0 @@ -// Module included in the following assemblies: -// -// * machine_management/creating-infrastructure-machinesets.adoc -// * logging/cluster-logging-moving.adoc - -:_mod-docs-content-type: PROCEDURE -[id="infrastructure-moving-logging_{context}"] -= Moving {logging} resources - -You can configure the {clo} to deploy the pods for {logging} components, such as Elasticsearch and Kibana, to different nodes. You cannot move the {clo} pod from its installed location. - -For example, you can move the Elasticsearch pods to a separate node because of high CPU, memory, and disk requirements. - -.Prerequisites - -* You have installed the {clo} and the {es-op}. - -.Procedure - -. Edit the `ClusterLogging` custom resource (CR) in the `openshift-logging` project: -+ -[source,terminal] ----- -$ oc edit ClusterLogging instance ----- -+ -.Example `ClusterLogging` CR -[source,yaml] ----- -apiVersion: logging.openshift.io/v1 -kind: ClusterLogging -# ... -spec: - logStore: - elasticsearch: - nodeCount: 3 - nodeSelector: <1> - node-role.kubernetes.io/infra: '' - tolerations: - - effect: NoSchedule - key: node-role.kubernetes.io/infra - value: reserved - - effect: NoExecute - key: node-role.kubernetes.io/infra - value: reserved - redundancyPolicy: SingleRedundancy - resources: - limits: - cpu: 500m - memory: 16Gi - requests: - cpu: 500m - memory: 16Gi - storage: {} - type: elasticsearch - managementState: Managed - visualization: - kibana: - nodeSelector: <1> - node-role.kubernetes.io/infra: '' - tolerations: - - effect: NoSchedule - key: node-role.kubernetes.io/infra - value: reserved - - effect: NoExecute - key: node-role.kubernetes.io/infra - value: reserved - proxy: - resources: null - replicas: 1 - resources: null - type: kibana -# ... ----- -<1> Add a `nodeSelector` parameter with the appropriate value to the component you want to move. You can use a `nodeSelector` in the format shown or use `: ` pairs, based on the value specified for the node. If you added a taint to the infrasructure node, also add a matching toleration. - -.Verification - -To verify that a component has moved, you can use the `oc get pod -o wide` command. - -For example: - -* You want to move the Kibana pod from the `ip-10-0-147-79.us-east-2.compute.internal` node: -+ -[source,terminal] ----- -$ oc get pod kibana-5b8bdf44f9-ccpq9 -o wide ----- -+ -.Example output -[source,terminal] ----- -NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES -kibana-5b8bdf44f9-ccpq9 2/2 Running 0 27s 10.129.2.18 ip-10-0-147-79.us-east-2.compute.internal ----- - -* You want to move the Kibana pod to the `ip-10-0-139-48.us-east-2.compute.internal` node, a dedicated infrastructure node: -+ -[source,terminal] ----- -$ oc get nodes ----- -+ -.Example output -[source,terminal] ----- -NAME STATUS ROLES AGE VERSION -ip-10-0-133-216.us-east-2.compute.internal Ready master 60m v1.25.0 -ip-10-0-139-146.us-east-2.compute.internal Ready master 60m v1.25.0 -ip-10-0-139-192.us-east-2.compute.internal Ready worker 51m v1.25.0 -ip-10-0-139-241.us-east-2.compute.internal Ready worker 51m v1.25.0 -ip-10-0-147-79.us-east-2.compute.internal Ready worker 51m v1.25.0 -ip-10-0-152-241.us-east-2.compute.internal Ready master 60m v1.25.0 -ip-10-0-139-48.us-east-2.compute.internal Ready infra 51m v1.25.0 ----- -+ -Note that the node has a `node-role.kubernetes.io/infra: ''` label: -+ -[source,terminal] ----- -$ oc get node ip-10-0-139-48.us-east-2.compute.internal -o yaml ----- -+ -.Example output -[source,yaml] ----- -kind: Node -apiVersion: v1 -metadata: - name: ip-10-0-139-48.us-east-2.compute.internal - selfLink: /api/v1/nodes/ip-10-0-139-48.us-east-2.compute.internal - uid: 62038aa9-661f-41d7-ba93-b5f1b6ef8751 - resourceVersion: '39083' - creationTimestamp: '2020-04-13T19:07:55Z' - labels: - node-role.kubernetes.io/infra: '' -... ----- - -* To move the Kibana pod, edit the `ClusterLogging` CR to add a node selector: -+ -[source,yaml] ----- -apiVersion: logging.openshift.io/v1 -kind: ClusterLogging -# ... -spec: -# ... - visualization: - kibana: - nodeSelector: <1> - node-role.kubernetes.io/infra: '' - proxy: - resources: null - replicas: 1 - resources: null - type: kibana ----- -<1> Add a node selector to match the label in the node specification. - -* After you save the CR, the current Kibana pod is terminated and new pod is deployed: -+ -[source,terminal] ----- -$ oc get pods ----- -+ -.Example output -[source,terminal] ----- -NAME READY STATUS RESTARTS AGE -cluster-logging-operator-84d98649c4-zb9g7 1/1 Running 0 29m -elasticsearch-cdm-hwv01pf7-1-56588f554f-kpmlg 2/2 Running 0 28m -elasticsearch-cdm-hwv01pf7-2-84c877d75d-75wqj 2/2 Running 0 28m -elasticsearch-cdm-hwv01pf7-3-f5d95b87b-4nx78 2/2 Running 0 28m -collector-42dzz 1/1 Running 0 28m -collector-d74rq 1/1 Running 0 28m -collector-m5vr9 1/1 Running 0 28m -collector-nkxl7 1/1 Running 0 28m -collector-pdvqb 1/1 Running 0 28m -collector-tflh6 1/1 Running 0 28m -kibana-5b8bdf44f9-ccpq9 2/2 Terminating 0 4m11s -kibana-7d85dcffc8-bfpfp 2/2 Running 0 33s ----- - -* The new pod is on the `ip-10-0-139-48.us-east-2.compute.internal` node: -+ -[source,terminal] ----- -$ oc get pod kibana-7d85dcffc8-bfpfp -o wide ----- -+ -.Example output -[source,terminal] ----- -NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES -kibana-7d85dcffc8-bfpfp 2/2 Running 0 43s 10.131.0.22 ip-10-0-139-48.us-east-2.compute.internal ----- - -* After a few moments, the original Kibana pod is removed. -+ -[source,terminal] ----- -$ oc get pods ----- -+ -.Example output -[source,terminal] ----- -NAME READY STATUS RESTARTS AGE -cluster-logging-operator-84d98649c4-zb9g7 1/1 Running 0 30m -elasticsearch-cdm-hwv01pf7-1-56588f554f-kpmlg 2/2 Running 0 29m -elasticsearch-cdm-hwv01pf7-2-84c877d75d-75wqj 2/2 Running 0 29m -elasticsearch-cdm-hwv01pf7-3-f5d95b87b-4nx78 2/2 Running 0 29m -collector-42dzz 1/1 Running 0 29m -collector-d74rq 1/1 Running 0 29m -collector-m5vr9 1/1 Running 0 29m -collector-nkxl7 1/1 Running 0 29m -collector-pdvqb 1/1 Running 0 29m -collector-tflh6 1/1 Running 0 29m -kibana-7d85dcffc8-bfpfp 2/2 Running 0 62s ----- - diff --git a/post_installation_configuration/cluster-tasks.adoc b/post_installation_configuration/cluster-tasks.adoc index dbb24b407335..306b2269b18b 100644 --- a/post_installation_configuration/cluster-tasks.adoc +++ b/post_installation_configuration/cluster-tasks.adoc @@ -629,7 +629,13 @@ include::modules/infrastructure-moving-registry.adoc[leveloffset=+2] include::modules/infrastructure-moving-monitoring.adoc[leveloffset=+2] -include::modules/infrastructure-moving-logging.adoc[leveloffset=+2] +[id="custer-tasks-moving-logging-resources"] +=== Moving {logging} resources + +For information about moving {logging} resources, see: + +* xref:../logging/scheduling_resources/logging-node-selectors.adoc#logging-node-selectors[Using node selectors to move logging resources] +* xref:../logging/scheduling_resources/logging-taints-tolerations.adoc#cluster-logging-logstore-tolerations_logging-taints-tolerations[Using taints and tolerations to control logging pod placement] include::modules/cluster-autoscaler-about.adoc[leveloffset=+1] include::modules/cluster-autoscaler-cr.adoc[leveloffset=+2]