MultiCluster Failover with ClusterAffinities Not Functioning well with WorkloadRebalancer

[surenthar-rajamohan-work]

You are welcome to discuss the use of this feature.

As mentioned here: #4990 by @RainbowMango

Good day, I saw that the previous related issue was closed with this comment:

This feature was included in the release-1.18. See https://github.com/karmada-io/karmada/blob/master/docs/CHANGELOG/CHANGELOG-1.18.md#overflow-cluster-affinities-for-hybrid-cloud-scheduling

However, I would like to raise a concern because the behavior I am seeing still does not seem to match what is described in the documentation for clusterAffinities, especially for the disaster recovery use case.

I am referring to this documentation:

https://karmada.io/docs/userguide/scheduling/propagation-policy/#use-case-2-disaster-recovery-primary-and-backup-clusters

Based on my understanding, with this configuration:

clusterAffinities:
  - affinityName: primary-cluster
    clusterNames:
      - member-cluster-1

  - affinityName: backup-cluster
    clusterNames:
      - member-cluster-2

Karmada should prefer member-cluster-1 first. If member-cluster-1 fails or cannot satisfy scheduling, it should move the workload to member-cluster-2. After member-cluster-1 recovers, I expected WorkloadRebalancer to trigger fresh scheduling and move the workload back to member-cluster-1, since it is the first affinity group.

However, I am seeing different behavior.

After failing over to member-cluster-2, I removed all taints from both clusters and confirmed that member-cluster-1 is healthy:

member-cluster-2 taints=
member-cluster-1 taints=

CompleteAPIEnablements=True reason=Complete message=collected complete APIEnablements from the cluster
Ready=True reason=ClusterReady message=cluster is healthy and ready to accept workloads

Then I applied a WorkloadRebalancer using the correct Deployment name. The WorkloadRebalancer completed successfully:

Status:
  Finish Time:          2026-07-03T02:46:31Z
  Observed Generation:  1
  Observed Workloads:
    Result:  Successful
    Workload:
      API Version:  apps/v1
      Kind:         Deployment
      Name:         example-deployment
      Namespace:    example-namespace

The ResourceBinding also shows that rescheduling was triggered:

Reschedule Triggered At:  2026-07-03T02:46:31Z
Last Scheduled Time:     2026-07-03T02:46:31Z

However, after the successful rebalance, the workload still remained on the backup cluster:

selected=[{"name":"member-cluster-2","replicas":1}]
observingAffinity=backup-cluster

The ResourceBinding event also shows:

ScheduleBindingSucceed
Binding has been scheduled successfully. Result: {member-cluster-2:1}

So the rebalancer seems to work, but the scheduler still selects member-cluster-2 even though member-cluster-1 is listed first in clusterAffinities, is Ready=True, and has no taints.

My Deployment PropagationPolicy:

apiVersion: policy.karmada.io/v1alpha1
kind: PropagationPolicy
metadata:
  name: example-deployment-pp
  namespace: example-namespace
spec:
  conflictResolution: Overwrite
  propagateDeps: true
  priority: 1

  resourceSelectors:
    - apiVersion: apps/v1
      kind: Deployment
      labelSelector:
        matchLabels:
          app.kubernetes.io/instance: example-app

  placement:
    clusterAffinities:
      - affinityName: primary-cluster
        clusterNames:
          - member-cluster-1

      - affinityName: backup-cluster
        clusterNames:
          - member-cluster-2

    spreadConstraints:
      - spreadByField: cluster
        minGroups: 1
        maxGroups: 1

    replicaScheduling:
      replicaSchedulingType: Duplicated

  failover:
    cluster:
      purgeMode: Directly

My WorkloadRebalancer:

apiVersion: apps.karmada.io/v1alpha1
kind: WorkloadRebalancer
metadata:
  name: example-failback-to-primary
spec:
  ttlSecondsAfterFinished: 600
  workloads:
    - apiVersion: apps/v1
      kind: Deployment
      name: example-deployment
      namespace: example-namespace

My prerequisite PropagationPolicy, which keeps the prerequisites in both clusters:

apiVersion: policy.karmada.io/v1alpha1
kind: PropagationPolicy
metadata:
  name: example-prereq-pp
  namespace: example-namespace
spec:
  conflictResolution: Overwrite
  propagateDeps: true
  priority: 10

  resourceSelectors:
    - apiVersion: v1
      kind: Namespace
      name: example-namespace

    - apiVersion: v1
      kind: ServiceAccount
      labelSelector:
        matchLabels:
          app.kubernetes.io/instance: example-app

    - apiVersion: v1
      kind: ConfigMap
      labelSelector:
        matchLabels:
          app.kubernetes.io/instance: example-app

    - apiVersion: bitnami.com/v1alpha1
      kind: SealedSecret
      labelSelector:
        matchLabels:
          app.kubernetes.io/instance: example-app

  placement:
    clusterAffinity:
      clusterNames:
        - member-cluster-1
        - member-cluster-2

    clusterTolerations:
      - operator: Exists

The current ResourceBinding for the Deployment shows this:

Spec:
  Clusters:
    Name:               member-cluster-2
    Replicas:           1

  Placement:
    Cluster Affinities:
      Affinity Name:  primary-cluster
      Cluster Names:
        member-cluster-1
      Affinity Name:  backup-cluster
      Cluster Names:
        member-cluster-2

  Reschedule Triggered At:  2026-07-03T02:46:31Z

Status:
  Aggregated Status:
    Applied:       true
    Cluster Name:  member-cluster-2
    Health:        Healthy

  Conditions:
    Message:  Binding has been scheduled successfully.
    Reason:   Success
    Status:   True
    Type:     Scheduled

  Last Scheduled Time:                2026-07-03T02:46:31Z
  Scheduler Observed Generation:      28
  Scheduler Observing Affinity Name:  backup-cluster

The event confirms the final scheduling result:

Normal  ScheduleBindingSucceed  default-scheduler  Binding has been scheduled successfully. Result: {member-cluster-2:1}

My question is:

Is this expected behavior for clusterAffinities?

Specifically, after the primary cluster recovers and WorkloadRebalancer successfully triggers rescheduling, should the workload move back to the first matching affinity group, primary-cluster, or is clusterAffinities only used for fallback during failure and not for automatic/manual failback?

If the documented disaster recovery example requires additional configuration to support failback to the primary cluster, could you please clarify what is missing from my configuration?

---

Additional version information:

I also checked the running Karmada component versions and confirmed that this does not seem to be an unsupported-field issue.

The scheduler reports:

karmada-scheduler version: version.Info{
  GitVersion:"v1.18.0-40-g7edc6e691",
  GitCommit:"7edc6e6919cc0207007383ebd5078ca3a4fbc13d",
  GitShortCommit:"7edc6e691",
  GitTreeState:"clean",
  BuildDate:"2026-06-17T09:15:24Z",
  GoVersion:"go1.26.4",
  Compiler:"gc",
  Platform:"linux/arm64"
}

The controller-manager reports:

karmada-controller-manager version: version.Info{
  GitVersion:"v1.18.0-66-gffbade988",
  GitCommit:"ffbade988fc3d6652daa504ff6cac37141e9f755",
  GitShortCommit:"ffbade988",
  GitTreeState:"clean",
  BuildDate:"2026-06-30T03:08:13Z",
  GoVersion:"go1.26.4",
  Compiler:"gc",
  Platform:"linux/amd64"
}

The webhook reports:

karmada-webhook version: version.Info{
  GitVersion:"v1.18.0-66-gffbade988",
  GitCommit:"ffbade988fc3d6652daa504ff6cac37141e9f755",
  GitShortCommit:"ffbade988",
  GitTreeState:"clean",
  BuildDate:"2026-06-30T03:09:36Z",
  GoVersion:"go1.26.4",
  Compiler:"gc",
  Platform:"linux/arm64"
}

The API also exposes overflowAffinities:

kubectl explain propagationpolicy.spec.placement.clusterAffinities.overflowAffinities

FIELD: overflowAffinities <[]Object>

DESCRIPTION:
  OverflowAffinities defines additional cluster groups that the scheduler
  can progressively include when the primary group has insufficient resources.

So the cluster appears to be running a build newer than v1.18.0, and the CRD/API supports the newer affinity fields. However, WorkloadRebalancer still does not move the workload back to the first affinity group after the primary cluster becomes healthy again.