Rackspace Support Request: Pods Not Scheduling in tt-secondary-cluster

[turnalfred]

Summary

We need help remediating a cluster-level scheduling issue in tt-secondary-cluster.

As of July 8, 2026, newly created pods are remaining in Pending indefinitely and are not being assigned to any node. This affects unrelated workloads in multiple namespaces and suggests a control-plane scheduler issue rather than an application-specific problem.

Impact

• New workload pods are not being scheduled
• Scheduled jobs and backup jobs are not starting
• Affected namespaces include:
  • bonarda-budget-sync-dev
  • databases

At the time of investigation there were 15 unscheduled pods stuck in Pending.

What We Observed

• The active kubectl context was pointed at tt-secondary-cluster
• Worker nodes were Ready
• Sampled worker node had:
  • no taints
  • Unschedulable: false
  • low-to-moderate allocated requests only
• ResourceQuota objects: none found
• LimitRange objects: none found
• Pending pods had:
  • Node: <none>
  • no PodScheduled condition
  • no scheduler events
  • no workload-specific constraint explaining the failure

We also created a fresh test pod (busybox) in the default namespace on July 8, 2026. It also remained Pending with:

• Node: <none>
• no scheduling events
• no PodScheduled condition

That reproduced the issue on a brand-new pod and confirms this is not isolated to one application or namespace.

Control Plane Evidence

• kube-scheduler lease exists in kube-system
• Lease was still renewing during investigation
• Example observed renew time:
  • 2026-07-08T09:07:07Z

This suggests the scheduler process appears alive, but it is not acting on unscheduled pods.

Timing

Based on current cluster evidence, the scheduling issue appears to have started before July 8, 2026 and most likely began on July 1, 2026.

The best current approximation from cluster data is:

• scheduling was still working for databases/production-pg-cluster-backup-29714400-bc4c5, which was created at 2026-07-01T00:00:00Z and completed successfully
• the oldest currently unscheduled pod is bonarda-budget-sync-dev/bonarda-budget-sync-backend-dev-secondary-bonarda-budget-s5tsbj, created at 2026-07-01T11:02:27Z

This places the likely start of the issue in the window:

• between 2026-07-01T00:00:00Z and 2026-07-01T11:02:27Z

If a single approximate start time is needed for support triage, the safest cluster-based estimate is:

• approximately the morning of July 1, 2026 UTC

The unscheduled pods on this cluster span multiple creation times, including:

• 2026-07-01T11:02:27Z
• 2026-07-01T12:14:03Z
• 2026-07-02T00:00:00Z
• 2026-07-02T02:54:40Z
• 2026-07-03T15:39:24Z
• 2026-07-04T06:26:28Z
• 2026-07-05T04:24:09Z
• 2026-07-06T17:08:55Z
• 2026-07-07T22:43:00Z
• 2026-07-08T05:53:40Z

This indicates the issue has been affecting newly created pods on tt-secondary-cluster for several days, including pods created on July 8, 2026.

Cleanup Already Performed

• No Evicted pods were present during investigation
• A temporary busybox test pod was created to reproduce the issue and then removed

Requested Rackspace Assistance

Please investigate and remediate the scheduler / control-plane path for tt-secondary-cluster.

Specifically, please check for:

• scheduler health beyond lease renewal
• scheduler watch / informer failures
• API server to scheduler event delivery issues
• any managed control-plane degradation affecting pod scheduling
• any provider-side control-plane errors affecting this cluster over the July 1 to July 8, 2026 period

Desired Outcome

• Restore normal pod scheduling in tt-secondary-cluster
• Confirm root cause
• Advise whether the existing 15 stuck Pending pods should be manually deleted and recreated after remediation

Example Affected Pods

• bonarda-budget-sync-dev/bonarda-budget-sync-backend-dev-secondary-bonarda-budget-sc8qk9
• bonarda-budget-sync-dev/bonarda-budget-sync-backend-dev-secondary-bonarda-budget-s5tsbj
• bonarda-budget-sync-dev/bonarda-budget-sync-backend-dev-secondary-bonarda-budget-szwd2n
• databases/production-pg-cluster-backup-29715840-g6pn6
• databases/production-pg-cluster-backup-29715840-g86hk

[turnalfred]

@vjha-pf9 @sirishkr Is any one here to Support on these??

[vjha-pf9]

Cluster is healthy now and all required pods are in Running status.

[sirishkr]

@turnalfred we're seeing you're repeatedly running into these cluster scheduling problems that we don't see reported as much elsewhere. I have filed #444 to track resolution of the root cause. Please use that issue going forward to help us keep our work on this problem organized.