GPU and AI ML

GPU and AI-ML

Clusters that train and serve models add a layer of operators on top of Kubernetes: device drivers that hand out GPUs, a queue that decides which job may start, and platforms that run Ray and model servers. Sextant has sidebar sections for these, with each operator's own status explained in plain words, the links between objects, and problem categories. They work like the Ecosystem views: a section appears only when the cluster serves its kinds, everything is read-only (the usual Edit and Delete are the only actions), and everything comes from the Kubernetes API.

Contents: Dynamic Resource Allocation · NVIDIA GPU Operator · Kueue · JobSet · KubeRay · KServe · Problems · What Sextant does not do

Dynamic Resource Allocation

Section Dynamic Resource Allocation (resource.k8s.io): Resource Claims, Claim Templates, Device Classes, Resource Slices, and a Devices page. It answers three questions: which pods use which claims, what was allocated where, and why a pod waits for a claim.

• Every API version. The resource.k8s.io API has changed shape four times (v1alpha3, v1beta1, v1beta2, v1). Sextant reads each object by the fields it has, not by its version, so the same views work on all of them.
• A claim's state is Pending (nothing allocated), Allocated, In use (a pod is reserved for it), Binding failed (the driver reported a binding failure on the allocated device) or Deleting. The allocation lists the devices, the driver and pool, and the node when the allocation is pinned to exactly one; any other node selector is shown as text.
• Requests are shown as written, including the ordered "first available" alternatives. A pod's details list its claims, direct or made from a template.
• Why a pod waits. A Pending pod that cannot be scheduled shows the scheduler's own words, with the DRA reasons recognised: it cannot allocate all claims, a claim is not created yet from its template, a device class does not exist, a claim is in use. Events are not watched.
• Devices. A node's details list the devices its drivers publish. The Devices page lists every device of every Resource Slice (at most 500 slices) and which claim holds it. If the claim list is cut short or cannot be read, "allocated to" says "cannot tell", never "free".
• CEL is never evaluated. Device classes and requests select devices with CEL expressions. Sextant shows them as written, with a note, and never says which devices a selector would match: only the scheduler can say that, and a second evaluator would disagree with it in exactly the cases you are debugging. Driver configuration is shown as JSON text.

NVIDIA GPU Operator

Section NVIDIA GPU Operator (nvidia.com): Cluster Policies, NVIDIA Drivers and a GPU nodes page.

• State is the operator's own: Ready or Not ready, with the condition's reason and message.
• Components. A Cluster Policy's details list each component the operator deploys and whether it is rolled out: driver, container toolkit, device plugin, DCGM and its exporter, GPU Feature Discovery, MIG manager, node status exporter, the operator validator, and the sandbox and Kata components when sandbox workloads are on. A component is expected only when the Cluster Policy enables it (the operator's defaults apply when it says nothing), and is found by the DaemonSet name the operator gives it in the namespace named in the policy's status. With Use NVIDIA Driver CRD on, the driver row points to the NVIDIA Driver objects and the DaemonSets they own.
• MIG. The policy's MIG strategy is shown; an unset one reads "not set" (the device plugin and GPU Feature Discovery then decide, and the default is none).
• GPU nodes are the nodes labelled nvidia.com/gpu.present=true (at most 500), with the GPU product, count and memory, the MIG configuration and its state, and the GPU and nvidia.com/mig-* capacity and allocatable. A Node's details get a GPU block from the node object alone.

Kueue

Section Kueue (kueue.x-k8s.io): Cluster Queues, Local Queues, Resource Flavors, Workloads, Admission Checks, Cohorts and Workload Priority Classes. Both v1beta1 and v1beta2 are read, whichever the cluster serves.

• A queue's state is its Active condition. Stopped (you set a stop policy) and Terminating are not faults. Any other reason (flavor not found, admission check not found, cohort cycle…) is shown as inactive with Kueue's message.
• Quota against usage. A Cluster Queue's details show, per flavor and resource, the nominal quota beside what is reserved (with the part borrowed from the cohort, and the share of the nominal quota, which goes over 100 % while borrowing) and what is in use by admitted workloads, plus the borrowing and lending limits. A Cohort shows its own quotas, without usage.
• Why a Workload waits. A Workload's state comes from its conditions, in Kueue's order: Finished, Deactivated, Admitted, Evicted, Waiting for admission checks, Pending, or Inadmissible (misconfigured). The reason it waits is the message the scheduler wrote ("couldn't assign flavors to pod set main: insufficient unused quota for cpu in flavor …"), shown as Kueue wrote it. A Workload that waits because its queue is stopped or inactive is told through the queue. Pod sets show their requests and assigned flavors, and the evictions so far.
• A queue's waiting Workloads. A Cluster Queue lists the waiting Workloads of the Local Queues that point at it, oldest first (at most 50), with how long and why; a Local Queue lists its own. Past 5000 Workloads the list says it may be incomplete.
• The Queue section on jobs. The details of a Job, JobSet, RayJob, RayCluster, Kubeflow job, TrainJob, LeaderWorkerSet or AppWrapper get a Queue section with the job's Workload and its state: queued, waiting and why, admitted, evicted. The Workload is found by the label Kueue puts on it (kueue.x-k8s.io/job-uid), and a prebuilt Workload by the name the job gives. A job without the queue label shows the section only if a Workload exists for it. Nothing is watched where Kueue is not installed.

JobSet

Section JobSet (jobset.x-k8s.io): JobSets.

• State is Completed, Failed, Suspended, Restarting, Starting (startup policy in progress), Running or Pending, from the JobSet's own conditions.
• Restarts. The details show the restarts that count toward the failure policy's limit against maxRestarts; a JobSet without a failure policy has no limit. Controllers older than v0.6 counted every restart, and Sextant counts that way for them.
• Why it failed. The failed JobSet's message names its first failed Job; Sextant links it and shows that Job's own failure reason and message (backoff limit exceeded, pod failure policy…).
• Jobs and pods. While the details are open, the JobSet's Jobs and pods are listed by the JobSet label (at most 500 each, pods that are not running first, at most 50 shown).

KubeRay

Section KubeRay (ray.io): Ray Clusters, Ray Jobs and Ray Services, explained from the operator's own status; both the newer condition-based status and the older state fields are read.

• Ray Cluster: Ready, Provisioning, Head not ready, Failed, Suspending or Suspended. The details list its pods and count the ready ones per worker group; for an autoscaling group with no upper limit they say "at least N, no upper limit".
• Ray Job: Pending, Running, Retrying, Succeeded, Failed, Stopped or Suspended. A failed attempt that will be retried is Retrying, not failed.
• Ray Service: Ready, Not ready, Upgrading, Failed or Suspended, with the Serve applications' status and messages. During an upgrade the active cluster keeps serving; a rollback reads as "Rollback in progress".
• Links. A Ray Cluster links to its head pod and Service; a Ray Job to its Ray Cluster and, in Kubernetes-job mode, its submitter Job; a Ray Service to its active and pending Ray Clusters and its serve Service.
• Logs. Head logs opens the head pod's log; Worker logs opens the merged log of all worker pods (the usual limit of 20 streams in one merged tab); Submitter logs opens the merged log of the Ray Job's submitter pods.
• What is masked. A Ray Service's Serve config (serveConfigV2) is shown as YAML with every value under env_vars and every value under a secret-looking key (token, key, auth, secret, password, credential, API or access key; not max_tokens or tokenizer) replaced by <hidden>, and credentials removed from URLs. A Ray Job's entrypoint has secret-looking --flag=value, --flag value and NAME=value parts masked the same way. A text that cannot be read safely is not shown ("see YAML"). The Ray Job's runtime environment (runtimeEnvYAML) is not shown at all. The YAML section remains the deliberate place to see the object as stored. Text from the operator (messages that can quote a job's last log lines) has URL credentials removed too.

KServe

Section KServe (serving.kserve.io): Inference Services, Serving Runtimes, Cluster Serving Runtimes, Inference Graphs and Trained Models.

• An Inference Service's state: Ready, Not ready, Progressing, Stopped (the stop annotation or condition) or Failed (the model failed to load, or the spec is invalid). The explanation is the model's own failure message, else the most basic failing condition: the components (predictor, transformer, explainer) before configurations, deployments, routes and ingress.
• Components and deployment mode. The details show the predictor, transformer and explainer with their URLs and revisions, and the deployment mode (Knative or Standard, the old names Serverless and RawDeployment read as the new). When neither the status nor an annotation says, it reads "the cluster's default": Sextant does not read KServe's configuration. What KServe made for the service is linked: Knative Services (in the Knative section), Deployments and Services.
• Canary. The traffic split between revisions of a component is shown, with the canary share and links to the Knative Revisions.
• Runtime choice. The details say which Serving Runtime KServe uses for the model and why, by KServe's own rules: a runtime named in the spec (a namespace one first, then a cluster one) must be enabled and support the model's format and protocol; otherwise the namespace's runtimes, then the cluster's, are filtered (enabled, the right model format and version among auto-selectable formats, the protocol, multi-node exactly when the predictor has workers) and sorted by KServe's order (protocol version, newest, name, then the format's priority). The first wins. You also see the other candidates in order, or why none fits, and a note when the choice differs from the runtime KServe recorded in the status. If a list cannot be read in full, the answer is "cannot be told", not a guess.
• URIs without credentials. Every URI shown (storage URIs, status URLs, component and graph step URLs, trained model storage) is shown without user name, password, query and fragment, and "credentials and query hidden" says when something was removed. The same holds for URIs inside condition and failure messages. A pre-signed URL's signature is never printed; the YAML section shows the object as stored.

Problems

All of these categories are on Settings → General → Notifications → Notify about and on the Problems and notifications page. Only Kueue Workloads is off by default.

Category	What counts
Dynamic Resource Allocation	A pod stuck Pending for 5 minutes because a claim it needs is not allocated, does not exist or was not created yet (a pod held back by scheduling gates is not judged, and an unallocated claim counts only when the scheduler names a DRA reason). Only on clusters that have a Device Class
NVIDIA GPU Operator	A Cluster Policy or NVIDIA Driver not ready for 10 minutes; an enabled component whose pods are not ready or not updated for 10 minutes; a node whose MIG configuration state is failed for 2 minutes
Kueue queues	A Cluster Queue inactive for another reason than a stop, and a Local Queue whose Cluster Queue does not exist, after 2 minutes
Kueue Workloads (off by default)	A Workload waiting for quota for 30 minutes, inadmissible for 5 minutes, evicted 3 times or more, or deactivated by Kueue for a failure (requeuing limit, maximum execution time, a rejected admission check; not a deactivation you made)
JobSet	A failed JobSet for a day, with the controller's reason and message; a running one with at most one restart left before maxRestarts
KubeRay	A Ray Cluster failed (after 5 minutes for a replica failure, at once for the old failed state) or with its head not ready for 5 minutes; a failed Ray Job for a day; a Ray Service not ready for 10 minutes (at once when its spec is invalid or it timed out initializing), or reporting an unhealthy Serve application for 5 minutes
KServe	An Inference Service whose model failed to load, at once; one with an invalid spec, after 2 minutes; one not ready for 10 minutes (a very large model that is still downloading can notify; the model status says so)

Stopped, suspended and deleting objects have none. Kueue Workloads is opt-in because it watches every Workload of the cluster (at most 5000), and a Workload holds its job's whole pod template; turn it on with Kueue Workloads waiting or evicted on the Problems page. The Cluster Queue waiting lists work while their details are open even with the category off.

None of these categories are given to AI agents by get_problems, whether or not they are turned on.

What Sextant does not do

• It installs, configures or upgrades none of these projects, and starts, stops, evicts, requeues or deactivates nothing. Use the projects' own tools, or change the object with Edit.
• It does not evaluate DRA's CEL selectors, does not decide which devices a request could get, and does not recompute whether a Workload fits a quota: the scheduler's and Kueue's own verdicts are shown.
• It does not read KServe's configuration, ModelMesh's own placement, Ray's dashboard, or metrics; nothing is sent to a model server or a Ray cluster.
• AI agents and extensions get nothing new from these views.

Next: Ecosystem views, Autoscaling, Problems and notifications, Using Sextant.