Kueue migration 1.1 integration

_attributes/common-attributes.adoc

@@ -79,7 +79,10 @@ endif::[]
 :descheduler-operator: Kube Descheduler Operator
 :cli-manager: CLI Manager Operator
 :lws-operator: Leader Worker Set Operator
-:kueue-prod-name: Red{nbsp}Hat build of Kueue
+//Kueue
+:kueue-name: Red{nbsp}Hat build of Kueue
+:kueue-op: Red Hat Build of Kueue Operator
+:ms: Red{nbsp}Hat build of MicroShift (MicroShift)
 // Backup and restore
 :launch: image:app-launcher.png[title="Application Launcher"]
 :mtc-first: Migration Toolkit for Containers (MTC)

_topic_maps/_topic_map.yml

@@ -3432,6 +3432,34 @@ Distros: openshift-enterprise
 Topics:
 - Name: Overview of AI workloads on OpenShift Container Platform
   File: index
+- Name: Red Hat build of Kueue
+  Dir: kueue
+  Distros: openshift-enterprise
+  Topics:
+  - Name: Introduction to Red Hat build of Kueue
+    File: about-kueue
+  - Name: Release notes
+    File: release-notes
+  - Name: Installing Red Hat build of Kueue
+    File: install-kueue
+  - Name: Installing Red Hat build of Kueue in a disconnected environment
+    File: install-disconnected
+  - Name: Configuring role-based permissions
+    File: rbac-permissions
+  - Name: Configuring quotas
+    File: configuring-quotas
+  - Name: Managing jobs and workloads
+    File: managing-workloads
+  - Name: Using cohorts
+    File: using-cohorts
+  - Name: Configuring fair sharing
+    File: configuring-fairsharing
+  - Name: Gang scheduling
+    File: gangscheduling
+  - Name: Running jobs with quota limits
+    File: running-kueue-jobs
+  - Name: Getting support
+    File: getting-support
 - Name: Leader Worker Set Operator
   Dir: leader_worker_set
   Distros: openshift-enterprise

ai_workloads/index.adoc

@@ -15,6 +15,7 @@ include::modules/ai-operators.adoc[leveloffset=+1]
 [role="_additional-resources"]
 .Additional resources
 
+* xref:../ai_workloads/kueue/about-kueue.adoc#about-kueue[Introduction to {kueue-name}]
 * xref:../ai_workloads/leader_worker_set/index.adoc#lws-about[{lws-operator} overview]
 
 // Exclude this for now until we can get it reviewed by the RHOAI team

ai_workloads/kueue/_attributes

@@ -0,0 +1 @@
+../../_attributes/

ai_workloads/kueue/about-kueue.adoc

@@ -0,0 +1,52 @@
+:_mod-docs-content-type: ASSEMBLY
+include::_attributes/common-attributes.adoc[]
+[id="about-kueue"]
+= Introduction to {kueue-name}
+:context: about-kueue
+
+toc::[]
+
+{kueue-name} is a Kubernetes-native system that manages access to resources for jobs.
+{kueue-name} can determine when a job waits, is admitted to start by creating pods, or should be _preempted_, meaning that active pods for that job are deleted.
+
+[NOTE]
+====
+In the context of {kueue-name}, a job can be defined as a one-time or on-demand task that runs to completion.
+====
+
+{kueue-name} is based on the link:https://kueue.sigs.k8s.io/docs/[Kueue] open source project.
+
+{kueue-name} is compatible with environments that use heterogeneous, elastic resources. This means that the environment has many different resource types, and those resources are capable of dynamic scaling.
+
+{kueue-name} does not replace any existing components in a Kubernetes cluster, but instead integrates with the existing Kubernetes API server, scheduler, and cluster autoscaler components.
+
+{kueue-name} supports all-or-nothing semantics. This means that either an entire job with all of its components is admitted to the cluster, or the entire job is rejected if it does not fit on the cluster.
+
+// Personas
+[id="about-kueue-personas"]
+== Personas
+
+Different personas exist in a {kueue-name} workflow.
+
+Batch administrators:: Batch administrators manage the cluster infrastructure and establish quotas and queues.
+Batch users:: Batch users run jobs on the cluster. Examples of batch users might be researchers, AI/ML engineers, or data scientists.
+Serving users:: Serving users run jobs on the cluster. For example, to expose a trained AI/ML model for inference.
+Platform developers:: Platform developers integrate {kueue-name} with other software. They might also contribute to the Kueue open source project.
+
+[id="about-kueue-workflow"]
+== Workflow overview
+
+The {kueue-name} workflow can be described at a high level as follows:
+
+. Batch administrators create and configure `ResourceFlavor`, `LocalQueue`, and `ClusterQueue` resources.
+. User personas create jobs on the cluster.
+. The Kubernetes API server validates and accepts job data.
+. {kueue-name} admits jobs based on configured options, such as order or quota. It injects affinity into the job by using resource flavors, and creates a `Workload` object that corresponds to each job.
+. The applicable controller for the job type creates pods.
+. The Kubernetes scheduler assigns pods to a node in the cluster.
+. The Kubernetes cluster autoscaler provisions more nodes as required.
+
+////
+TODO:Add docs explaining different job / workload types
+These can be added as we add stories / docs for different use cases
+////

ai_workloads/kueue/configuring-fairsharing.adoc

@@ -0,0 +1,18 @@
+:_mod-docs-content-type: ASSEMBLY
+include::_attributes/common-attributes.adoc[]
+[id="configuring-fairsharing"]
+= Configuring fair sharing
+:context: configuring-fairsharing
+
+toc::[]
+
+Fair sharing is a preemption strategy that is used to achieve an equal or weighted share of borrowable resources between the tenants of a cohort. Borrowable resources are the unused nominal quota of all the cluster queues in a cohort.
+
+You can configure fair sharing by setting the `preemptionPolicy` value in the `Kueue` custom resource (CR) to `FairSharing`.
+
+include::modules/kueue-clusterqueue-share-value.adoc[leveloffset=+1]
+
+[role="_additional-resources"]
+[id="additional-resources_{context}"]
+== Additional resources
+* xref:../../ai_workloads/kueue/install-kueue.adoc#create-kueue-cr_install-kueue[Creating a `Kueue` custom resource]

ai_workloads/kueue/configuring-quotas.adoc

@@ -0,0 +1,38 @@
+:_mod-docs-content-type: ASSEMBLY
+include::_attributes/common-attributes.adoc[]
+[id="configuring-quotas"]
+= Configuring quotas
+:context: configuring-quotas
+
+toc::[]
+
+As an administrator, you can use {kueue-name} to configure quotas to optimize resource allocation and system throughput for user workloads.
+You can configure quotas for compute resources such as CPU, memory, pods, and GPU.
+
+You can configure quotas in {kueue-name} by completing the following steps:
+
+. Configure a cluster queue.
+. Configure a resource flavor.
+. Configure a local queue.
+
+Users can then submit their workloads to the local queue.
+
+include::modules/kueue-configuring-clusterqueues.adoc[leveloffset=+1]
+
+[role="_next-steps"]
+[id="clusterqueues-next-steps_{context}"]
+.Next steps
+
+The cluster queue is not ready for use until a xref:../../ai_workloads/kueue/configuring-quotas.adoc#configuring-resourceflavors_configuring-quotas[`ResourceFlavor` object] has also been configured.
+
+include::modules/kueue-configuring-resourceflavors.adoc[leveloffset=+1]
+
+include::modules/kueue-configuring-localqueues.adoc[leveloffset=+1]
+
+include::modules/kueue-configuring-localqueue-defaults.adoc[leveloffset=+1]
+
+[role="_additional-resources"]
+[id="clusterqueues-additional-resources_{context}"]
+== Additional resources
+* xref:../../ai_workloads/kueue/rbac-permissions.adoc#rbac-permissions[RBAC permissions]
+* link:https://kueue.sigs.k8s.io/docs/concepts/cluster_queue/[Kubernetes documentation about cluster queues]

ai_workloads/kueue/gangscheduling.adoc

@@ -0,0 +1,27 @@
+:_mod-docs-content-type: ASSEMBLY
+include::_attributes/common-attributes.adoc[]
+[id="gangscheduling"]
+= Gang scheduling
+:context: gangscheduling
+
+toc::[]
+
+Gang scheduling ensures that a group or _gang_ of related jobs only start when all required resources are available. {kueue-name} enables gang scheduling by suspending jobs until the {product-title} cluster can guarantee the capacity to start and execute all of the related jobs in the gang together. This is also known as _all-or-nothing_ scheduling.
+
+Gang scheduling is important if you are working with expensive, limited resources, such as GPUs. Gang scheduling can prevent jobs from claiming but not using GPUs, which can improve GPU utilization and can reduce running costs. Gang scheduling can also help to prevent issues like resource segmentation and deadlocking.
+
+include::modules/kueue-configuring-gangscheduling.adoc[leveloffset=+1]
+
+[role="_additional-resources"]
+[id="additional-resources_{context}"]
+== Additional resources
+* xref:../../ai_workloads/kueue/install-kueue.adoc#create-kueue-cr_install-kueue[Creating a Kueue custom resource]
+
+////
+// use case - deep learning
+One classic example is in deep learning workloads. Deep learning frameworks (Tensorflow, PyTorch etc) require all the workers to be running during the training process.
+
+In this scenario, when you deploy training workloads, all the components should be scheduled and deployed to ensure the training works as expected.
+
+Gang Scheduling is a critical feature for Deep Learning workloads to enable all-or-nothing scheduling capability, as most DL frameworks requires all workers to be running to start training process. Gang Scheduling avoids resource inefficiency and scheduling deadlock sometimes.
+////

ai_workloads/kueue/getting-support.adoc

@@ -0,0 +1,27 @@
+:_mod-docs-content-type: ASSEMBLY
+include::_attributes/common-attributes.adoc[]
+[id="getting-support"]
+= Getting support
+:context: getting-support
+
+toc::[]
+
+If you experience difficulty with a procedure described in this documentation, or with {kueue-name} in general, visit the link:http://access.redhat.com[Red{nbsp}Hat Customer Portal].
+
+From the Customer Portal, you can:
+
+* Search or browse through the Red{nbsp}Hat Knowledgebase of articles and solutions relating to Red{nbsp}Hat products.
+* Submit a support case to Red{nbsp}Hat Support.
+* Access other product documentation.
+
+[id="getting-support-rh-kb"]
+== About the Red Hat Knowledgebase
+
+The link:https://access.redhat.com/knowledgebase[Red{nbsp}Hat Knowledgebase] provides rich content aimed at helping you make the most of Red{nbsp}Hat's products and technologies. The Red{nbsp}Hat Knowledgebase consists of articles, product documentation, and videos outlining best practices on installing, configuring, and using Red{nbsp}Hat products. In addition, you can search for solutions to known issues, each providing concise root cause descriptions and remedial steps.
+
+include::modules/kueue-gathering-cluster-data.adoc[leveloffset=+1]
+
+[id="getting-support-additional-resources"]
+[role="_additional-resources"]
+== Additional resources
+* xref:../../support/index.adoc#support-overview[Support overview]

ai_workloads/kueue/images

@@ -0,0 +1 @@
+../../images/

ai_workloads/kueue/install-disconnected.adoc

@@ -0,0 +1,31 @@
+:_mod-docs-content-type: ASSEMBLY
+include::_attributes/common-attributes.adoc[]
+[id="install-disconnected"]
+= Installing {kueue-name} in a disconnected environment
+:context: install-disconnected
+
+toc::[]
+
+Before you can install {kueue-name} on a disconnected {product-title} cluster, you must enable {olm-first} in disconnected environments by completing the following steps:
+
+* Disable the default remote OperatorHub sources for OLM.
+* Use a workstation with full internet access to create and push local mirrors of the OperatorHub content to a mirror registry.
+* Configure OLM to install and manage Operators from local sources on the mirror registry instead of the default remote sources.
+
+After enabling OLM in a disconnected environment, you can continue to use your unrestricted workstation to keep your local OperatorHub sources updated as newer versions of Operators are released.
+
+For full documentation on completing these steps, see the {product-title} documentation on xref:../../disconnected/using-olm.adoc#olm-restricted-networks[Using Operator Lifecycle Manager in disconnected environments].
+
+include::modules/kueue-compatible-environments.adoc[leveloffset=+1]
+
+include::modules/kueue-install-kueue-operator.adoc[leveloffset=+1]
+
+[role="_additional-resources"]
+.Additional resources
+* xref:../../security/cert_manager_operator/cert-manager-operator-install.adoc#installing-the-cert-manager-operator-for-red-hat-openshift[Installing the {cert-manager-operator}]
+
+include::modules/upgrading-kueue.adoc[leveloffset=+1]
+
+include::modules/kueue-create-kueue-cr.adoc[leveloffset=+1]
+
+include::modules/kueue-label-namespaces.adoc[leveloffset=+1]

ai_workloads/kueue/install-kueue.adoc

@@ -0,0 +1,23 @@
+:_mod-docs-content-type: ASSEMBLY
+include::_attributes/common-attributes.adoc[]
+[id="install-kueue"]
+= Installing {kueue-name}
+:context: install-kueue
+
+toc::[]
+
+You can install {kueue-name} by using the {kueue-op} in OperatorHub.
+
+include::modules/kueue-compatible-environments.adoc[leveloffset=+1]
+
+include::modules/kueue-install-kueue-operator.adoc[leveloffset=+1]
+
+[role="_additional-resources"]
+.Additional resources
+* xref:../../security/cert_manager_operator/cert-manager-operator-install.adoc#installing-the-cert-manager-operator-for-red-hat-openshift[Installing the {cert-manager-operator}]
+
+include::modules/upgrading-kueue.adoc[leveloffset=+1]
+
+include::modules/kueue-create-kueue-cr.adoc[leveloffset=+1]
+
+include::modules/kueue-label-namespaces.adoc[leveloffset=+1]

ai_workloads/kueue/managing-workloads.adoc

@@ -0,0 +1,13 @@
+:_mod-docs-content-type: ASSEMBLY
+include::_attributes/common-attributes.adoc[]
+[id="managing-workloads"]
+= Managing jobs and workloads
+:context: managing-workloads
+
+toc::[]
+
+{kueue-name} does not directly manipulate jobs that are created by users. Instead, Kueue manages `Workload` objects that represent the resource requirements of a job. {kueue-name} automatically creates a workload for each job, and syncs any decisions and statuses between the two objects.
+
+include::modules/kueue-label-namespaces.adoc[leveloffset=+1]
+
+include::modules/kueue-configuring-labelpolicy.adoc[leveloffset=+1]

ai_workloads/kueue/modules

@@ -0,0 +1 @@
+../../modules/

ai_workloads/kueue/rbac-permissions.adoc

@@ -0,0 +1,26 @@
+:_mod-docs-content-type: ASSEMBLY
+include::_attributes/common-attributes.adoc[]
+[id="rbac-permissions"]
+= Configuring role-based permissions
+:context: rbac-permissions
+
+toc::[]
+
+The following procedures provide information about how you can configure role-based access control (RBAC) for your {kueue-name} deployment. These RBAC permissions determine which types of users can create which types of {kueue-name} objects.
+
+[id="authentication-clusterroles"]
+== Cluster roles
+
+The {kueue-name} Operator deploys `kueue-batch-admin-role` and `kueue-batch-user-role` cluster roles by default.
+
+kueue-batch-admin-role:: This cluster role includes the permissions to manage cluster queues, local queues, workloads, and resource flavors.
+kueue-batch-user-role:: This cluster role includes the permissions to manage jobs and to view local queues and workloads.
+
+include::modules/kueue-configure-rbac-batch-admins.adoc[leveloffset=+1]
+
+include::modules/kueue-configure-rbac-batch-users.adoc[leveloffset=+1]
+
+[role="_additional-resources"]
+== Additional resources
+* xref:../../authentication/using-rbac.adoc#using-rbac[Using RBAC to define and apply permissions]
+* xref:../../authentication/index.adoc#openshift-auth-common-terms_overview-of-authentication-authorization[Glossary of common terms for {product-title} authentication and authorization]

ai_workloads/kueue/release-notes.adoc

@@ -0,0 +1,17 @@
+:_mod-docs-content-type: ASSEMBLY
+include::_attributes/common-attributes.adoc[]
+[id="release-notes"]
+= Release notes
+:context: release-notes
+
+toc::[]
+
+{kueue-name} is released as an Operator that is supported on {product-title}.
+
+include::modules/kueue-compatible-environments.adoc[leveloffset=+1]
+
+include::modules/kueue-release-notes-1.1.adoc[leveloffset=+1]
+
+include::modules/kueue-release-notes-1.0.1.adoc[leveloffset=+1]
+
+include::modules/kueue-release-notes-1.0.adoc[leveloffset=+1]

ai_workloads/kueue/running-kueue-jobs.adoc

@@ -0,0 +1,13 @@
+:_mod-docs-content-type: ASSEMBLY
+include::_attributes/common-attributes.adoc[]
+[id="running-kueue-jobs"]
+= Running jobs with quota limits
+:context: running-kueue-jobs
+
+toc::[]
+
+You can run Kubernetes jobs with {kueue-name} enabled to manage resource allocation within defined quota limits. This can help to ensure predictable resource availability, cluster stability, and optimized performance.
+
+include::modules/kueue-identifying-local-queues.adoc[leveloffset=+1]
+
+include::modules/kueue-defining-running-jobs.adoc[leveloffset=+1]

ai_workloads/kueue/snippets

@@ -0,0 +1 @@
+../../snippets/

ai_workloads/kueue/troubleshooting.adoc

@@ -0,0 +1,31 @@
+:_mod-docs-content-type: ASSEMBLY
+include::_attributes/common-attributes.adoc[]
+[id="troubleshooting"]
+= Troubleshooting
+:context: troubleshooting
+
+toc::[]
+
+// commented out - note for TS docs
+
+// Troubleshooting installations
+// Verifying node health
+// Troubleshooting network issues
+// Troubleshooting Operator issues
+// Investigating pod issues
+// Diagnosing CLI issues
+////
+Troubleshooting Jobs
+Troubleshooting the status of a Job
+
+Troubleshooting Queues
+Troubleshooting the status of a LocalQueue or ClusterQueue
+
+Troubleshooting Provisioning Request in Kueue
+Troubleshooting the status of a Provisioning Request in Kueue
+
+Troubleshooting Pods
+Troubleshooting the status of a Pod or group of Pods
+
+Troubleshooting delete ClusterQueue
+////