Rackspace Support Request: Pods Not Scheduling in tlc-pp-cluster

[turnalfred]

Summary

We need help remediating a cluster-level scheduling issue in tlc-pp-cluster.

As of July 8, 2026, newly created pods are remaining in Pending indefinitely and are not being assigned to any node. This affects multiple namespaces and unrelated workloads, which suggests a control-plane scheduler issue rather than an application-specific problem.

Impact

• New workload pods are not being scheduled.
• Scheduled jobs and backup jobs are not starting.
• Affected namespaces include:
  • bonarda-dsa-dev
  • databases
  • gitlab-system
  • klokin-klokaut-dev
  • klokin-klokaut-prod
  • nexus-proxy-dev

At the time of investigation there were 17 unscheduled pods stuck in Pending.

What We Observed

• The active kubectl context was pointed at tlc-pp-cluster.
• Worker nodes were Ready.
• Sampled worker node had:
  • no taints
  • Unschedulable: false
  • moderate allocated requests only
• ResourceQuota objects: none found
• LimitRange objects: none found
• Pending pods had:
  • Node: <none>
  • no PodScheduled condition
  • no scheduler events
  • no workload-specific constraint explaining the failure

We also created a fresh test pod (busybox) in the default namespace on July 8, 2026. It also remained Pending with:

• Node: <none>
• no scheduling events
• no PodScheduled condition

That reproduced the issue on a brand-new pod and confirms this is not isolated to one application or namespace.

Control Plane Evidence

• kube-scheduler lease exists in kube-system
• Lease was still renewing during investigation
• Example observed renew time:
  • 2026-07-08T08:56:32Z

This suggests the scheduler process appears alive, but it is not acting on unscheduled pods.

Timing

The unscheduled pods started accumulating on July 8, 2026, beginning around:

• 2026-07-08T00:00:00Z
• 2026-07-08T00:37:16Z
• 2026-07-08T01:00:00Z
• 2026-07-08T03:00:00Z
• 2026-07-08T05:00:00Z
• 2026-07-08T05:30:00Z

This looks like the issue began around the start of July 8 UTC and then continued affecting all new pods created afterward.

Cleanup Already Performed

We cleaned up stale evicted pods during triage:

• 68 Evicted pods deleted
• Verification after cleanup showed 0 remaining pods with status.reason=Evicted

This cleanup did not resolve the scheduling problem.

Requested Rackspace Assistance

Please investigate and remediate the scheduler / control-plane path for tlc-pp-cluster.

Specifically, please check for:

• scheduler health beyond lease renewal
• scheduler watch / informer failures
• API server to scheduler event delivery issues
• any managed control-plane degradation affecting pod scheduling
• any provider-side control-plane errors beginning on July 8, 2026

Desired Outcome

• Restore normal pod scheduling in tlc-pp-cluster
• Confirm root cause
• Advise whether the existing 17 stuck Pending pods should be manually deleted and recreated after remediation

Example Affected Pods

• bonarda-dsa-dev/bonarda-dsa-backend-dev-streak-reset-29724480-srbv8
• databases/dev-pg-cluster-backup-29724480-dn8g2
• gitlab-system/tt-gitlab-toolbox-backup-29724480-2n56p
• klokin-klokaut-prod/klokin-klokaut-backend-scheduler-prod-daily-check-29724540ns5zh
• nexus-proxy-dev/nexus-proxy-blocklist-scan-29724480-jcxp2

[turnalfred]

@vjha-pf9 @sirishkr Is any one here to Support on these??

[vjha-pf9]

Cluster is healthy now and all required pods are in Running status.