Skip to content

docs: enhance ManifestWorkReplicaSet documentation with rollout mechanism #524

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Oct 23, 2025
Merged

docs: enhance ManifestWorkReplicaSet documentation with rollout mechanism #524

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
277 changes: 204 additions & 73 deletions content/en/docs/concepts/work-distribution/manifestworkreplicaset.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,69 @@ aliases:

## What is `ManifestWorkReplicaSet`

`ManifestWorkReplicaSet` is an aggregator API that uses [Manifestwork](https://github.com/open-cluster-management-io/open-cluster-management-io.github.io/blob/main/content/en/concepts/manifestwork.md) and [Placement](https://github.com/open-cluster-management-io/open-cluster-management-io.github.io/blob/main/content/en/concepts/placement.md) to create manifestwork for the placement-selected clusters.
`ManifestWorkReplicaSet` is an aggregator API that uses [ManifestWork](https://open-cluster-management.io/docs/concepts/work-distribution/manifestwork/) and [Placement](https://open-cluster-management.io/docs/concepts/content-placement/placement/) to automatically create `ManifestWork` resources for clusters selected by `Placement`. It simplifies multi-cluster workload distribution by eliminating the need to manually create individual ManifestWork resources for each target cluster.

View an example of `ManifestWorkReplicaSet` to deploy a CronJob and Namespace for a group of clusters selected by placements.
## Feature Enablement

`ManifestWorkReplicaSet` is in **alpha release** and is not enabled by default. To enable this feature, you must configure the cluster-manager instance on the hub cluster.

Edit the cluster-manager CR:

```shell
$ oc edit ClusterManager cluster-manager
```

Add the `workConfiguration` field to enable the ManifestWorkReplicaSet feature gate:

```yaml
kind: ClusterManager
metadata:
name: cluster-manager
spec:
...
workConfiguration:
featureGates:
- feature: ManifestWorkReplicaSet
mode: Enable
```

Verify the feature is enabled successfully:

```shell
$ oc get ClusterManager cluster-manager -o yaml
```

Confirm that the `cluster-manager-work-controller` deployment appears in the status under `status.generations`:

```yaml
kind: ClusterManager
metadata:
name: cluster-manager
spec:
...
status:
...
generations:
...
- group: apps
lastGeneration: 2
name: cluster-manager-work-webhook
namespace: open-cluster-management-hub
resource: deployments
version: v1
- group: apps
lastGeneration: 1
name: cluster-manager-work-controller
namespace: open-cluster-management-hub
resource: deployments
version: v1
```

## Overview

Here's a simple example to get started with `ManifestWorkReplicaSet`.

This example deploys a `CronJob` and `Namespace` to a group of clusters selected by `Placement`.

```yaml
apiVersion: work.open-cluster-management.io/v1alpha1
Expand All @@ -25,22 +85,6 @@ spec:
- name: placement-rollout-all # Name of a created Placement
rolloutStrategy:
type: All
- name: placement-rollout-progressive # Name of a created Placement
rolloutStrategy:
type: Progressive
progressive:
minSuccessTime: 5m
progressDeadline: 10m
maxFailures: 5%
mandatoryDecisionGroups:
- groupName: "prod-canary-west"
- groupName: "prod-canary-east"
- name: placement-rollout-progressive-per-group # Name of a created Placement
rolloutStrategy:
type: ProgressivePerGroup
progressivePerGroup:
progressDeadline: 10m
maxFailures: 2
manifestWorkTemplate:
deleteOption:
propagationPolicy: SelectivelyOrphan
Expand Down Expand Up @@ -91,13 +135,150 @@ spec:
- '-c'
- date; echo Hello from the Kubernetes cluster
```
The **PlacementRefs** uses the Rollout Strategy [API](https://github.com/open-cluster-management-io/api/blob/main/cluster/v1alpha1/types_rolloutstrategy.go) to apply the manifestWork to the selected clusters.
In the example above; the placementRefs refers to three placements; placement-rollout-all, placement-rollout-progressive and placement-rollout-progressive-per-group. For more info regarding the rollout strategies check the Rollout Strategy section at the [placement](https://github.com/open-cluster-management-io/open-cluster-management-io.github.io/blob/main/content/en/concepts/placement.md) document.
**Note:** The placement reference must be in the same namespace as the manifestWorkReplicaSet.
The `placementRefs` field uses the Rollout Strategy [API](https://github.com/open-cluster-management-io/api/blob/main/cluster/v1alpha1/types_rolloutstrategy.go) to control how `ManifestWork` resources are applied to the selected clusters.

In the example above, the `placementRefs` references `placement-rollout-all` with a `rolloutStrategy` of `All`, which means the workload will be deployed to all selected clusters simultaneously.

## Rollout Strategy

`ManifestWorkReplicaSet` supports three rollout strategy types to control how workloads are distributed across clusters. For detailed rollout strategy documentation, see the [Placement rollout strategy section](https://open-cluster-management.io/docs/concepts/content-placement/placement/#rollout-strategy).

**Note:** The placement reference must be in the same namespace as the `ManifestWorkReplicaSet`.

1. **All**: Deploy to all selected clusters simultaneously
```yaml
placementRefs:
- name: placement-rollout-all # Name of a created Placement
rolloutStrategy:
type: All
```

2. **Progressive**: Gradual rollout with configurable parameters
```yaml
placementRefs:
- name: placement-rollout-progressive # Name of a created Placement
rolloutStrategy:
type: Progressive
progressive:
minSuccessTime: 5m
progressDeadline: 10m
maxFailures: 5%
mandatoryDecisionGroups:
- groupName: "prod-canary-west"
- groupName: "prod-canary-east"
```

3. **ProgressivePerGroup**: Progressive rollout per decision group
```yaml
placementRefs:
- name: placement-rollout-progressive-per-group # Name of a created Placement
rolloutStrategy:
type: ProgressivePerGroup
progressivePerGroup:
progressDeadline: 10m
maxFailures: 2
```

## Rollout Mechanism

The `ManifestWorkReplicaSet` rollout process is based on the `Progressing` and `Degraded` conditions of each `ManifestWork`. These conditions are not built-in but must be defined using [Custom CEL Expressions](https://open-cluster-management.io/docs/concepts/work-distribution/manifestwork/#custom-cel-expressions) in the `manifestConfigs` section.

### Condition Requirements

- **Progressing condition (required)**: Tracks whether the ManifestWork is currently being applied to the cluster. The rollout controller uses this condition to determine if a cluster deployment is in progress or completed.
- For rollout strategies `Progressive` and `ProgressivePerGroup`, this is a **required** condition to determine if the rollout can proceed to the next cluster.
- When the `Progressing` condition is `False`, the deployment is considered complete and succeeded, and the rollout will continue to the next cluster based on `minSuccessTime`.
- If this condition is not defined, the rollout will never proceed to the next cluster (it will remain stuck waiting for the condition).

- **Degraded condition (optional)**: Indicates if the ManifestWork has failed or encountered issues.
- This is an **optional** condition used only to determine failure states.
- If the `Degraded` condition status is `True`, the rollout may stop or count towards `maxFailures`.
- If this condition is not defined, the workload is assumed to never be degraded.

**Rollout Status Determination:**

The rollout controller determines the status of each ManifestWork based on the combination of `Progressing` and `Degraded` condition values:

| Progressing | Degraded | Rollout Status | Description |
|-------------|----------|----------------|-------------|
| `True` | `True` | **Failed** | Work is progressing but degraded |
| `True` | `False` or not set | **Progressing** | Work is being applied and is healthy |
| `False` | `False` or not set | **Succeeded** | Work has been successfully applied |
| Unknown/Not set | Any | **Progressing** | Conservative fallback: treat as still progressing |

### Example: Progressive Rollout with Custom Conditions

This example demonstrates a progressive rollout with custom CEL expressions that define `Progressing` and `Degraded` conditions for a Deployment:

```yaml
apiVersion: work.open-cluster-management.io/v1alpha1
kind: ManifestWorkReplicaSet
metadata:
name: mwrset
namespace: default
spec:
placementRefs:
- name: placement-test # Name of a created Placement
rolloutStrategy:
type: Progressive
progressive:
minSuccessTime: 1m
progressDeadline: 10m
maxFailures: 5%
maxConcurrency: 1
manifestWorkTemplate:
workload:
manifests:
- apiVersion: apps/v1
kind: Deployment
metadata:
name: hello
namespace: default
spec:
selector:
matchLabels:
app: hello
template:
metadata:
labels:
app: hello
spec:
containers:
- name: hello
image: busybox
command:
["sh", "-c", 'echo "Hello, Kubernetes!" && sleep 36000']
manifestConfigs:
- resourceIdentifier:
group: apps
resource: deployments
namespace: default
name: hello
conditionRules:
- condition: Progressing
type: CEL
celExpressions:
- |
!(
(object.metadata.generation == object.status.observedGeneration) &&
has(object.status.conditions) &&
object.status.conditions.exists(c, c.type == 'Progressing' && c.status == 'True' && c.reason == 'NewReplicaSetAvailable')
)
messageExpression: |
result ? "Applying" : "Completed"
- condition: Degraded
type: CEL
celExpressions:
- |
(object.metadata.generation == object.status.observedGeneration) &&
(has(object.status.readyReplicas) && has(object.status.replicas) && object.status.readyReplicas < object.status.replicas)
messageExpression: |
result ? "Degraded" : "Healthy"
```

## Status tracking

The ManifestWorkReplicaSet example above refers to three placements each one will have its placementSummary in ManifestWorkReplicaSet status. The PlacementSummary shows the number of manifestWorks applied to the placement's clusters based on the placementRef's rolloutStrategy and total number of clusters.
The PlacementSummary shows the number of manifestWorks applied to the placement's clusters based on the placementRef's rolloutStrategy and total number of clusters.
The manifestWorkReplicaSet Summary aggregate the placementSummaries showing the total number of applied manifestWorks to all clusters.

The manifestWorkReplicaSet has three status conditions;
Expand Down Expand Up @@ -173,54 +354,4 @@ status:
progressing: 0
degraded: 0
total: 45
```
## Release and Enable Feature

ManifestWorkReplicaSet is in alpha release and it is not enabled by default. In order to enable the ManifestWorkReplicaSet feature, it has to be enabled in the cluster-manager instance in the hub. Use the following command to edit the cluster-manager CR (custom resource) in the hub cluster.

```shell
$ oc edit ClusterManager cluster-manager
```
Add the workConfiguration field to the cluster-manager CR as below and save.

```yaml
kind: ClusterManager
metadata:
name: cluster-manager
spec:
...
workConfiguration:
featureGates:
- feature: ManifestWorkReplicaSet
mode: Enable
```
In order to ensure the ManifestWorkReplicaSet has been enabled successfully, check the cluster-manager using the command below

```shell
$ oc get ClusterManager cluster-manager -o yml
```
You should find under the status->generation the cluster-manager-work-controller deployment has been added as below

```yaml
kind: ClusterManager
metadata:
name: cluster-manager
spec:
...
status:
...
generations:
...
- group: apps
lastGeneration: 2
name: cluster-manager-work-webhook
namespace: open-cluster-management-hub
resource: deployments
version: v1
- group: apps
lastGeneration: 1
name: cluster-manager-work-controller
namespace: open-cluster-management-hub
resource: deployments
version: v1
```
```