Skip to content

Test Results

Jump to bottom
Sandeep Bazar edited this page Jul 26, 2026 · 1 revision

ocm-mcp-server - end-to-end test report

Verdict: 🟢 ALL GREEN  ·  generated 2026-07-26 08:41:29  ·  local kind fleet via podman, Open Cluster Management hub

Passed Expected-unavailable Failed Total steps
53 1 0 57

Reproduce locally with ./hack/e2e-local.sh - one command stands up a real multi-cluster fleet on kind, drives every tool and prompt, breaks a workload and fixes it through the approval gate, then tears the fleet down.

What this end-to-end test does

One command stands up a real multi-cluster Open Cluster Management fleet on kind, then drives the MCP server through every tool and prompt: it reads the fleet freely, but forces every change through policy checks and a human-approved token. It then deliberately breaks a workload and shows the agent diagnose and fix it, before tearing the fleet down. Green means the whole loop worked.

  1. 🧰 Install & verify tools - podman, kind, kubectl, clusteradm, helm
  2. ☸️ Bootstrap a real fleet - 1 hub + N spoke clusters, OCM, Kyverno, policies
  3. ✨ Enrich & seed fixtures - add-ons, policies, ManifestWorks - so every tool has real data
  4. 🔍 Investigate (read tools) - inventory, health, placement, work, add-ons - free, no gate
  5. 🛡️ Make a change, gated - propose → Kyverno + guardrails → human token → apply (a bad token is refused)
  6. 💬 Prompts & audit - reusable runbooks and an append-only audit trail
  7. 💥 Break something - inject a failing rollout (ImagePullBackOff)
  8. 🩺 Diagnose - health + events pinpoint the bad image
  9. 🔧 Fix, through the gate - propose the pinned image, approve, apply
  10. ✅ Verify recovery - re-read health until the workload is healthy again
  11. 🧹 Clean up - delete only the clusters this run created

1. Dependencies

✅ OK podman

Required tool for the local fleet.

$ podman --version

already installed: podman version 6.0.2

✅ OK kind

Required tool for the local fleet.

$ kind version

already installed: kind v0.32.0 go1.26.3 darwin/arm64

✅ OK kubectl

Required tool for the local fleet.

$ kubectl version --client=true -o yaml 2>/dev/null | grep gitVersion | head -1

already installed:   gitVersion: v1.34.1

✅ OK helm

Required tool for the local fleet.

$ helm version --short

already installed: v4.1.1+g5caf004

✅ OK clusteradm

Required tool for the local fleet.

$ clusteradm version | head -1

already installed: clusteradm	version	:v1.3.1-0-g90bdc31

✅ OK oc

Required tool for the local fleet.

$ oc version --client 2>/dev/null | head -1

already installed: Client Version: 4.18.0-202504282205.p0.geb9bc9b.assembly.stream.el9-eb9bc9b

2. Python package

✅ OK pip install -e .

Install the MCP server and its ocm-mcp CLI into an isolated virtualenv.

$ pip install -e .

ocm-mcp-server 0.2.1

3. Container engine

✅ OK podman running

kind needs a container runtime; here it is Podman (Docker is not required).

$ podman info

podman version 6.0.2

4. Bootstrap fleet

✅ OK bootstrap.sh

Stand up a real OCM hub with spokes so every tool talks to genuine clusters, not mocks.

$ CONTAINER_ENGINE=podman SPOKES=2 ./hack/bootstrap.sh --no-jaeger


[1;36m==> Deploying the demo app to all spokes (namespace 'shop')[0m
namespace/shop created
deployment.apps/payments created
service/payments created
namespace/shop created
deployment.apps/payments created
service/payments created

[1;36m==> Done. Configure your MCP client environment:[0m

  export OCM_MCP_HUB_CONTEXT=kind-hub
  export OCM_MCP_SPOKE_CONTEXTS=cluster1=kind-cluster1,cluster2=kind-cluster2
  export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318   # if Jaeger enabled

  # then register the server with your MCP client, e.g. for Claude Code:
  #   see examples/claude-code.mcp.json

Verify the fleet:
  kubectl --context kind-hub get managedclusters

✅ OK managed clusters

Proof the spokes are registered and Available on the hub.

$ kubectl --context kind-hub get managedclusters

NAME       HUB ACCEPTED   MANAGED CLUSTER URLS                  JOINED   AVAILABLE   AGE
cluster1   true           https://cluster1-control-plane:6443   True     True        104s
cluster2   true           https://cluster2-control-plane:6443   True     True        80s

4b. Enrich fleet

✅ OK add-on, feature gate, and API fixtures

Install the governance policy add-on, turn on ManifestWorkReplicaSet, and add minimal CRDs plus labelled sample objects for the ACM (ManagedClusterInfo) and HyperShift (HostedCluster/NodePool) APIs - so the add-on, policy, rollout, ACM, and HyperShift tools all return real objects. On a real hub those come from ACM/MCE or HyperShift; on kind they are clearly-labelled e2e fixtures.

$ clusteradm install/enable hub-addon; kubectl patch clustermanager; kubectl apply CRDs + samples

enrichment: governance-policy-framework(enabled) ManifestWorkReplicaSet(enabled) acm+hypershift-fixtures

4c. Setup fixtures - create the objects the read tools will report on

✅ OK Placement + ClusterSetBinding

Bind the 'global' cluster set into a namespace and add a Placement that selects every cluster - so the placement tools have a real decision to show.

$ kubectl apply Placement + ManagedClusterSetBinding

binding=created, placement=created

⚠️ SKIP seed ManifestWork

Deploy a marker ManifestWork so the work tools report on something real.

$ propose_manifestwork

REJECTED by hub admission (Kyverno policy):
{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Internal error occurred: failed calling webhook \"manifestworkvalidators.admission.work.open-cluster-management.io\": failed to call webhook: Post \"https://cluster-manager-work-webhook.open-cluster-management-hub.svc:9443/validate-work-open-cluster-management-io-v1-manifestwork?timeout=10s\": context deadline exceeded","reason":"InternalError","details":{"causes":[{"message":"failed calling webhook \"manifestworkvalidators.admission.work.open-cluster-management.io\": failed to call webhook: Post \"https://cluster-manager-work-webhook.open-cluster-management-hub.svc:9443/validate-work-open-cluster-management-io-v1-manifestwork?timeout=10s\": context deadline exceeded"}]},"code":500}

✅ OK ManifestWorkReplicaSet

A single template fanned across every placement-selected cluster - so list_manifestworkreplicasets shows a rollout.

$ kubectl apply ManifestWorkReplicaSet

result=created

✅ OK governance Policy

A sample inform Policy so the policy tools report on a real object.

$ kubectl apply Policy

result=created

5. Read tools (safe, no gate) - how an agent investigates a fleet

✅ OK list_clusters

The fleet roster: every managed cluster, its version and health.

$ tool: list_clusters

[
  {
    "name": "cluster1",
    "labels": {
      "cluster.open-cluster-management.io/clusterset": "default",
      "feature.open-cluster-management.io/addon-governance-policy-framework": "unreachable"
    },
    "available": "True",
    "joined": "True",
    "kubernetes_version": "v1.36.1",
    "capacity": {
      "cpu": "4",
      "memory": "8099496Ki"
    }
  },
  {
    "name": "cluster2",
    "labels": {
      "cluster.open-cluster-management.io/clusterset": "default",
      "feature.open-cluster-management.io/addon-governance-policy-framework": "unreachable"
    },
    "available": "True",
    "joined": "True",
    "kubernetes_version": "v1.36.1",
    "capacity": {
      "cpu": "4",
      "memory": "8099496Ki"
    }
  }
]

✅ OK get_cluster(cluster1)

Zoom into one cluster: acceptance, taints, capacity, claims.

$ tool: get_cluster

{
  "name": "cluster1",
  "labels": {
    "cluster.open-cluster-management.io/clusterset": "default",
    "feature.open-cluster-management.io/addon-governance-policy-framework": "unreachable"
  },
  "hub_accepts_client": true,
  "taints": [],
  "conditions": {
    "HubAcceptedManagedCluster": "True",
    "ManagedClusterJoined": "True",
    "ManagedClusterConditionAvailable": "True",
    "ManagedClusterConditionClockSynced": "True"
  },
  "kubernetes_version": "v1.36.1",
  "capacity": {
    "cpu": "4",
    "ephemeral-storage": "104266732Ki",
    "hugepages-1Gi": "0",
    "hugepages-2Mi": "0",
    "hugepages-32Mi": "0",
    "hugepages-64Ki": "0",
    "memory": "8099496Ki",
    "pods": "110"
  },
  "allocatable": {
    "cpu": "4",
    "ephemeral-storage": "104266732Ki",
    "hugepages-1Gi": "0",
    "hugepages-2Mi": "0",
    "hugepages-32Mi": "0",
    "hugepages-64Ki": "0",
    "memory": "8099496Ki",
    "pods": "110"
  },
  "cluster_claims": {}
}

✅ OK list_cluster_sets

Groupings of clusters used for placement.

$ tool: list_cluster_sets

[
  {
    "name": "default",
    "selector_type": "ExclusiveClusterSetLabel",
    "conditions": {
      "ClusterSetEmpty": "False"
    },
    "members": [
      "cluster1",
      "cluster2"
    ]
  },
  {
    "name": "global",
    "selector_type": "LabelSelector",
    "conditions": {
      "ClusterSetEmpty": "False"
    },
    "members": []
  }
]

✅ OK list_cluster_set_bindings

Which cluster sets a namespace may schedule to.

$ tool: list_cluster_set_bindings

[
  {
    "namespace": "default",
    "name": "global",
    "cluster_set": "global",
    "conditions": {
      "Bound": "True"
    }
  }
]

✅ OK list_cluster_claims

Facts each cluster self-reports (id, platform, version).

$ tool: list_cluster_claims

[
  {
    "cluster": "cluster1",
    "claims": {}
  },
  {
    "cluster": "cluster2",
    "claims": {}
  }
]

✅ OK get_cluster_health(cluster1)

The on-call view: unhealthy pods and degraded deployments.

$ tool: get_cluster_health

{
  "cluster": "cluster1",
  "hub_conditions": {
    "HubAcceptedManagedCluster": "True",
    "ManagedClusterJoined": "True",
    "ManagedClusterConditionAvailable": "True",
    "ManagedClusterConditionClockSynced": "True"
  },
  "unhealthy_pods": [],
  "degraded_deployments": [
    {
      "namespace": "shop",
      "name": "payments",
      "ready": "0/2"
    }
  ],
  "spoke_view": "ok"
}

✅ OK query_events(cluster1)

Recent Kubernetes events - the 'why' behind failures.

$ tool: query_events

[
  {
    "namespace": "shop",
    "type": "Normal",
    "reason": "Pulled",
    "object": "Pod/payments-df965d5f-6t7rx",
    "count": 1,
    "message": "Successfully pulled image \"registry.k8s.io/e2e-test-images/agnhost:2.47\" in 202ms (21.229s including waiting). Image size: 50637101 bytes."
  },
  {
    "namespace": "shop",
    "type": "Normal",
    "reason": "Created",
    "object": "Pod/payments-df965d5f-6t7rx",
    "count": 1,
    "message": "Container created"
  },
  {
    "namespace": "shop",
    "type": "Normal",
    "reason": "Started",
    "object": "Pod/payments-df965d5f-6t7rx",
    "count": 1,
    "message": "Container started"
  },
  {
    "namespace": "shop",
    "type": "Normal",
    "reason": "Pulled",
    "object": "Pod/payments-df965d5f-tghxp",
    "count": 1,
    "message": "Successfully pulled image \"registry.k8s.io/e2e-test-images/agnhost:2.47\" in 21.032s (21.032s including waiting). Image size: 50637101 bytes."
  },
  {
    "namespace": "shop",
    "type": "Normal",
    "reason": "Created",
    "object": "Pod/payments-df965d5f-tghxp",
    "count": 1,
    "message": "Container created"
... (15 more lines)

✅ OK list_manifestworks(cluster1)

What the hub is currently delivering to this cluster.

$ tool: list_manifestworks

[
  {
    "name": "demo-mwrstpl7f",
    "applied": "True",
    "available": "True",
    "resources": [
      "ConfigMap/e2e-seed (shop)"
    ]
  }
]

✅ OK list_manifestworkreplicasets

Fleet-wide rollouts (a template fanned across clusters).

$ tool: list_manifestworkreplicasets

[
  {
    "namespace": "default",
    "name": "demo-mwrs",
    "summary": {
      "applied": 2,
      "available": 2,
      "total": 2
    },
    "conditions": {
      "PlacementVerified": "True",
      "PlacementRolledOut": "False",
      "ManifestworkApplied": "True"
    }
  }
]

✅ OK list_cluster_management_addons

Fleet-level add-on definitions.

$ tool: list_cluster_management_addons

[
  {
    "name": "config-policy-controller",
    "install_strategy": "Manual",
    "conditions": {}
  },
  {
    "name": "governance-policy-framework",
    "install_strategy": "Manual",
    "conditions": {}
  }
]

✅ OK get_addon_health

Per-cluster add-on health across the fleet.

$ tool: get_addon_health

[
  {
    "cluster": "cluster1",
    "addon": "governance-policy-framework",
    "available": "Unknown",
    "degraded": "False",
    "progressing": "False"
  },
  {
    "cluster": "cluster2",
    "addon": "governance-policy-framework",
    "available": "Unknown",
    "degraded": "False",
    "progressing": "False"
  }
]

✅ OK list_addons_for_cluster(cluster1)

Every add-on installed on one cluster, with health.

$ tool: list_addons_for_cluster

[
  {
    "addon": "governance-policy-framework",
    "install_namespace": "open-cluster-management-agent-addon",
    "available": "Unknown",
    "degraded": "False"
  }
]

✅ OK list_pending_csrs

Clusters waiting to be admitted to the hub.

$ tool: list_pending_csrs

No pending CSRs - every cluster is already admitted (a healthy fleet).

✅ OK list_placements

Placements and how many clusters each selects.

$ tool: list_placements

[
  {
    "namespace": "default",
    "name": "demo-all",
    "cluster_sets": [
      "global"
    ],
    "number_of_clusters": null,
    "selected": 2,
    "conditions": {
      "PlacementMisconfigured": "False",
      "PlacementSatisfied": "True"
    }
  }
]

✅ OK get_placement_decision(demo-all)

Exactly which clusters a Placement chose.

$ tool: get_placement_decision

{
  "placement": "demo-all",
  "namespace": "default",
  "selected_clusters": [
    "cluster1",
    "cluster2"
  ],
  "count": 2
}

✅ OK list_policies

Governance Policies and per-cluster compliance.

$ tool: list_policies

[
  {
    "namespace": "default",
    "name": "require-cm",
    "remediation": "inform",
    "compliant": null,
    "per_cluster": {}
  }
]

✅ OK list_policy_violations

Only the NonCompliant / Pending policy-cluster pairs.

$ tool: list_policy_violations

No violations - evaluated policies are compliant (or evaluation is in progress).

✅ OK get_cluster_info(cluster1)

Extended inventory (OpenShift version, nodes) - ACM only.

$ tool: get_cluster_info

{
  "cluster": "cluster1",
  "console_url": "https://console-openshift-console.apps.cluster1.example.com",
  "kube_vendor": "OpenShift",
  "cloud_vendor": "BareMetal",
  "openshift_version": "4.16.7",
  "node_count": 1,
  "nodes": [
    {
      "name": "cluster1-control-plane",
      "capacity": {
        "cpu": "8",
        "memory": "16Gi"
      },
      "labels": {
        "node-role.kubernetes.io/control-plane": ""
      }
    }
  ],
  "conditions": {
    "ManagedClusterInfoSynced": "True"
  }
}

✅ OK list_hosted_clusters

HyperShift hosted control planes - when the hub hosts them.

$ tool: list_hosted_clusters

[
  {
    "name": "demo-hcp",
    "namespace": "clusters",
    "version": "4.16.7",
    "version_state": "Completed",
    "conditions": {
      "Available": "True"
    }
  }
]

✅ OK list_node_pools

HyperShift worker node pools - when the hub hosts HCPs.

$ tool: list_node_pools

[
  {
    "name": "demo-hcp-workers",
    "namespace": "clusters",
    "cluster": "demo-hcp",
    "desired_replicas": 2,
    "current_replicas": 2,
    "conditions": {
      "Ready": "True"
    }
  }
]

✅ OK list_resources(managedclusters)

Generic allow-listed reader over any OCM type.

$ tool: list_resources

[
  {
    "name": "cluster1",
    "labels": {
      "cluster.open-cluster-management.io/clusterset": "default",
      "feature.open-cluster-management.io/addon-governance-policy-framework": "unreachable"
    },
    "conditions": {
      "HubAcceptedManagedCluster": "True",
      "ManagedClusterJoined": "True",
      "ManagedClusterConditionAvailable": "True",
      "ManagedClusterConditionClockSynced": "True"
    }
  },
  {
    "name": "cluster2",
    "labels": {
      "cluster.open-cluster-management.io/clusterset": "default",
      "feature.open-cluster-management.io/addon-governance-policy-framework": "unreachable"
    },
    "conditions": {
      "HubAcceptedManagedCluster": "True",
      "ManagedClusterJoined": "True",
      "ManagedClusterConditionAvailable": "True",
      "ManagedClusterConditionClockSynced": "True"
    }
  }
]

✅ OK get_resource(managedclusters/cluster1)

Generic get of one allow-listed OCM object.

$ tool: get_resource

{
  "apiVersion": "cluster.open-cluster-management.io/v1",
  "kind": "ManagedCluster",
  "metadata": {
    "creationTimestamp": "2026-07-26T03:08:21Z",
    "finalizers": [
      "cluster.open-cluster-management.io/resource-cleanup",
      "cluster.open-cluster-management.io/api-resource-cleanup"
    ],
    "generation": 4,
    "labels": {
      "cluster.open-cluster-management.io/clusterset": "default",
      "feature.open-cluster-management.io/addon-governance-policy-framework": "unreachable"
    },
    "managedFields": [
      {
        "apiVersion": "cluster.open-cluster-management.io/v1",
        "fieldsType": "FieldsV1",
        "fieldsV1": {
          "f:spec": {
            "f:hubAcceptsClient": {}
          }
        },
        "manager": "clusteradm",
        "operation": "Update",
        "time": "2026-07-26T03:08:55Z"
      },
      {
        "apiVersion": "cluster.open-cluster-management.io/v1",
        "fieldsType": "FieldsV1",
        "fieldsV1": {
          "f:status": {
            ".": {},
            "f:allocatable": {
              ".": {},
              "f:cpu": {},
              "f:ephemeral-storage": {},
              "f:hugepages-1Gi": {},
              "f:hugepages-2Mi": {},
              "f:hugepages-32Mi": {},
... (7 more lines)

7. Gated write flow - propose, prove the gate rejects a bad token, then apply

✅ OK propose_manifestwork

The agent PROPOSES a change. It must pass static guardrails and a Kyverno dry-run first; nothing is applied yet.

$ tool: propose_manifestwork

proposal_id=fd619b573fc441468466ac07a1f46137
status=pending_approval

✅ PASS apply with a BAD token

Prove the gate holds: apply with an invalid token must be refused by the server, not applied.

$ tool: apply_manifestwork(bad token)

REJECTED: Malformed approval token.

✅ OK ocm-mcp approve (human)

A human on a trusted terminal mints an approval token: an Ed25519 signature binding the exact content and operation. The server holds only the public key, so it can verify a token but can never mint one - and neither can the agent.

$ ocm-mcp approve fd619b573fc441468466ac07a1f46137

token minted (...IvhtLnAIv0CQ)

✅ OK apply_manifestwork(token)

With a valid token, the server verifies it and delivers the change as an OCM ManifestWork.

$ tool: apply_manifestwork

{
  "status": "applied",
  "cluster": "cluster1",
  "manifestwork": "e2e-demo",
  "note": "Verify rollout with get_cluster_health / get_manifestwork. To undo, call propose_rollback then apply the rollback with a fresh approval."
}

✅ PASS get_manifestwork (verify)

Confirm the hub actually applied it on the spoke.

$ tool: get_manifestwork

{
  "cluster": "cluster1",
  "name": "e2e-demo",
  "conditions": {
    "Applied": "True",
    "Available": "True"
  },
  "resources": [
    {
      "resource": "ConfigMap/ocm-mcp-e2e",
      "namespace": "shop",
      "conditions": {
        "Applied": "True",
        "Available": "True",
        "StatusFeedbackSynced": "True"
      },
      "status_feedback": {}
    }
  ]
}

8. Gated lifecycle action - cordon a cluster out of scheduling, then undo

✅ OK propose_cluster_action(cordon)

Propose adding a NoSelect taint so Placements stop scheduling here - still gated by approval.

$ tool: propose_cluster_action

{
  "proposal_id": "f7f8b7c8f50c4b69890b152aec1907d3",
  "status": "pending_approval",
  "action": "cordon",
  "next_step": "Ask the human operator to run: ocm-mcp approve f7f8b7c8f50c4b69890b152aec1907d3 and provide you the approval token, then call apply_cluster_action."
}

✅ OK apply_cluster_action(cordon)

Applied after approval; the taint now keeps new work off this cluster.

$ tool: apply_cluster_action

taints=[{"effect": "NoSelect", "key": "ocm-mcp-server.io/cordoned", "timeAdded": "2026-07-26T03:10:39Z", "value": "true"}]

✅ OK uncordon (restore)

Remove the taint so the cluster is schedulable again.

$ tool: apply_cluster_action(uncordon)

taints_after=[]

9. Prompts - reusable runbooks the server hands any MCP client

✅ OK prompt: diagnose_fleet

A ready-made, safety-first runbook an agent can start from.

$ mcp prompt: diagnose_fleet

You are operating a Kubernetes fleet through the OCM hub. Investigate only - do not propose or apply anything yet.

1. Call list_clusters to see availability, version, and capacity of every cluster.
2. Call get_addon_health to spot Degraded or unavailable add-ons across the fleet.
3. For any cluster that is Unavailable or not joined, call get_cluster to read its conditions, then get_cluster_health for unhealthy pods and degraded deployments.
4. For each unhealthy workload, use query_events and get_pod_logs to find the cause.
5. Produce a concise report: per cluster, what is wrong, the evidence, and the smallest safe remediation you would propose. Do not act on it yet.

✅ OK prompt: remediate_with_approval

A ready-made, safety-first runbook an agent can start from.

$ mcp prompt: remediate_with_approval

A fleet operator reports: payments degraded on cluster1

Follow the safe remediation workflow:
1. Investigate with the read tools (list_clusters, get_cluster_health, query_events, get_pod_logs, get_manifestwork) until you can name the root cause with evidence.
2. Decide the smallest change that fixes it. For a workload change, call propose_manifestwork with pinned images and a precise summary. For a cluster lifecycle change (cordon, uncordon, set a label, accept a cluster), call propose_cluster_action.
3. If the proposal is REJECTED by static guardrails or Kyverno, read the reason, correct the manifest, and propose again. Never try to bypass a rejection.
4. Tell the operator the proposal id and ask them to run `ocm-mcp approve <id>` and give you the token. Wait for it.
5. Call the matching apply tool with the token. Then verify recovery with reads.
6. Finish by calling get_audit_trail an
... (truncated)

✅ OK prompt: why_not_scheduled

A ready-made, safety-first runbook an agent can start from.

$ mcp prompt: why_not_scheduled

Explain why cluster 'cluster1' was or was not selected by Placement 'demo-all' in namespace 'default'. Use only reads:

1. get_placement_decision('demo-all', 'default') for the clusters actually chosen.
2. list_placements to read the Placement's clusterSets and selection intent.
3. get_cluster('cluster1') for its labels, ClusterClaims, and taints.
4. list_cluster_set_bindings to confirm the Placement's namespace is bound to the ClusterSet the cluster belongs to.
5. list_addon_placement_scores('cluster1') if the Placement uses AddOn prioritizers.
6. Conclude with the specific reason: not in a bound ClusterSet, failed a predicate, carries a NoSelect taint, or simply out-scored by others.

10. Audit - the server's own record of every tool call

✅ OK list_pending_proposals

Proposals still awaiting human approval.

$ tool: list_pending_proposals

[
  {
    "id": "b5f924e6d7b4461cb049fc7d264fc059",
    "cluster": "cluster1",
    "kind": "action",
    "action": "uncordon",
    "name": "uncordon-cluster1",
    "summary": "Undo."
  },
  {
    "id": "b84e8f66ab8b4a4a9dc271f006674511",
    "cluster": "cluster1",
    "kind": "action",
    "action": "uncordon",
    "name": "uncordon-cluster1",
    "summary": "Undo."
  }
]

✅ OK get_audit_trail

The append-only log of every tool call so far - the agent writes its incident report from this record, not from memory.

$ tool: get_audit_trail

[
{"tool": "apply_manifestwork", "args": {"proposal_id": "fd619b573fc441468466ac07a1f46137", "approval_token": "<redacted>"}, "outcome": "rejected", "error": "", "duration_ms": 0, "ts": 1785035433.996419, "actor": "sandeepbazar:94984", "seq": 3, "prev": "31be6b299d6caead2e7b59a93afb8ba27c3cbafb41e9f2a3036071b8b45ba307", "hash": "b246ec932f37b8f53b3c7e92d8d7316cdca149012beeaf91fb8950d5c2edb130"},
{"tool": "apply_manifestwork", "args": {"proposal_id": "fd619b573fc441468466ac07a1f46137", "approval_token": "<redacted>"}, "outcome": "ok", "error": "", "duration_ms": 5, "ts": 1785035434.003003, "actor": "sandeepbazar:94984", "seq": 4, "prev": "b246ec932f37b8f53b3c7e92d8d7316cdca149012beeaf91fb8950d5c2edb130", "hash": "69428c0f7b28ca983b7caa06e630d0ef01a4f6c7d4ecaa7528799d1046df2fd1"},
{"tool": "get_manifestwork", "args": {"cluster": "cluster1", "name": "e2e-demo"}, "outcome": "ok", "error": "", "duration_ms": 4, "ts": 1785035439.0112379, "actor": "sandeepbazar:94984", "seq": 5, "prev": "69428c0f7b28ca983b7caa06e630d0ef01a4f6c7d4ecaa7528799d1046df2fd1", "hash": "bd2a558795ec107bf62aba5b9093204073dabb83f00b13074730b666b268bb25"},
{"tool": "propose_cluster_action", "args": {"cluster": "cluster1", "action": "cordon", "summary": "Cordon for maintenance.", "params_json": "{}"}, "outcome": "ok", "error": "", "duration_ms": 10, "ts": 1785035439.022474, "actor": "sandeepbazar:94984", "seq": 6
... (truncated)

12. Negative scenario - break something, then debug and fix it end to end

💥 INJECT INJECT: failing-rollout

Deploy a 'payments-v2' with a broken image tag, exactly the kind of bad rollout that pages someone at 2 a.m.

$ ./chaos/inject.sh failing-rollout cluster1

deployment.apps/payments-v2 created
injected: failing-rollout into cluster1

✅ OK DIAGNOSE: get_cluster_health

The agent asks 'what is unhealthy here?' and sees payments-v2 is not running.

$ tool: get_cluster_health(cluster1)

{
  "unhealthy_pods": [
    {
      "namespace": "shop",
      "name": "payments-v2-68bf75f6cc-c82rm",
      "phase": "Pending",
      "waiting": [
        "ErrImagePull"
      ],
      "restarts": 0
    },
    {
      "namespace": "shop",
      "name": "payments-v2-68bf75f6cc-xg557",
      "phase": "Pending",
      "waiting": [
        "ErrImagePull"
      ],
      "restarts": 0
    }
  ],
  "degraded_deployments": [
    {
      "namespace": "shop",
      "name": "payments-v2",
      "ready": "0/2"
    }
  ]
}

✅ OK DIAGNOSE: query_events

Events reveal the root cause in plain English: the image cannot be pulled.

$ tool: query_events(cluster1, shop)

[
  {
    "namespace": "shop",
    "type": "Normal",
    "reason": "BackOff",
    "object": "Pod/payments-v2-68bf75f6cc-c82rm",
    "count": 1,
    "message": "Back-off pulling image \"registry.k8s.io/e2e-test-images/agnhost:2.47-nonexistent\""
  },
  {
    "namespace": "shop",
    "type": "Warning",
    "reason": "Failed",
    "object": "Pod/payments-v2-68bf75f6cc-c82rm",
    "count": 1,
    "message": "Error: ImagePullBackOff"
  },
  {
    "namespace": "shop",
    "type": "Normal",
    "reason": "BackOff",
    "object": "Pod/payments-v2-68bf75f6cc-xg557",
    "count": 1,
    "message": "Back-off pulling image \"registry.k8s.io/e2e-test-images/agnhost:2.47-nonexistent\""
  },
  {
    "namespace": "shop",
    "type": "Warning",
    "reason": "Failed",
    "object": "Pod/payments-v2-68bf75f6cc-xg557",
    "count": 1,
    "message": "Error: ImagePullBackOff"
  },
  {
    "namespace": "shop",
    "type": "Warning",
    "reason": "Failed",
    "object": "Pod/payments-v2-68bf75f6cc-c82rm",
    "count": 1,
    "message": "Failed to pull image \"registry.k8s.io/e2e-test-images/agnhost:2.47-nonexistent\": rpc error: code = NotFound desc = failed to pull and unpack image \"registry.k8s.io/e2e-test-images/agnhost:2.47-nonexistent\": failed to resolve reference \"registry.k8s.io/e2e-test-images/agnhost:2.47-nonexistent\": registry.k8s.io/e2e-test-images/agnhost:2.47-nonexistent: not found
... (1 more lines)

🔧 FIX FIX: propose + approve

The agent proposes the smallest safe fix (a pinned, known-good image). It clears the guardrails and a human approves it.

$ propose_manifestwork -> ocm-mcp approve

proposal_id=952a811191664c83a311fbe1ce6310b5
guardrails=passed  kyverno_dry_run=passed  approved=yes

🔧 FIX FIX: apply_manifestwork

Delivered to the cluster through the hub as a ManifestWork.

$ tool: apply_manifestwork(token)

status=applied

✅ PASS VERIFY: recovery

Re-read health until payments-v2 is running again - the incident is closed, with a full audit trail of everything that happened.

$ tool: get_cluster_health(cluster1)

payments-v2 is Running again - fix confirmed end to end.

11. Cleanup

✅ OK teardown

Delete only the clusters created by this run (recorded in .e2e-run/created-clusters); never touch pre-existing clusters.

$ kind delete cluster --name ...

deleted: hub cluster1 cluster2

Code · LinkedIn · YouTube · Apache-2.0 · sponsorship via LinkedIn

ocm-mcp-server

The journey

Links

Clone this wiki locally