From daa59d11fd029c621b57b7c3ccf2b440d8a0c8ee Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E2=80=9CShauna=20Diaz=E2=80=9D?= Date: Tue, 14 Apr 2026 13:38:38 -0400 Subject: [PATCH] OSDOCS-18945: adds troubleshooting MCP gateway --- .../con-mcp-gateway-ts-gateway-routing.adoc | 29 ++++ .../proc-mcp-gateway-register-mcp-server.adoc | 5 + ...teway-ts-gateway-listener-not-working.adoc | 84 ++++++++++ ...proc-mcp-gateway-ts-pods-not-starting.adoc | 57 +++++++ ...way-ts-requests-fail-or-bypass-router.adoc | 152 ++++++++++++++++++ ...s-traffic-not-reaching-backend-server.adoc | 67 ++++++++ ...-ext-mcp-server-mcpserverregistration.adoc | 5 + .../mcp-gateway-troubleshooting.adoc | 13 +- 8 files changed, 411 insertions(+), 1 deletion(-) create mode 100644 modules/con-mcp-gateway-ts-gateway-routing.adoc create mode 100644 modules/proc-mcp-gateway-ts-gateway-listener-not-working.adoc create mode 100644 modules/proc-mcp-gateway-ts-pods-not-starting.adoc create mode 100644 modules/proc-mcp-gateway-ts-requests-fail-or-bypass-router.adoc create mode 100644 modules/proc-mcp-gateway-ts-traffic-not-reaching-backend-server.adoc diff --git a/modules/con-mcp-gateway-ts-gateway-routing.adoc b/modules/con-mcp-gateway-ts-gateway-routing.adoc new file mode 100644 index 000000000000..847d5bd38383 --- /dev/null +++ b/modules/con-mcp-gateway-ts-gateway-routing.adoc @@ -0,0 +1,29 @@ +// Module included in the following assemblies: +// +// *mcp_gateway_config/mcp-gateway-troubleshooting.adoc + +:_mod-docs-content-type: CONCEPT +[id="con-mcp-gateway-ts-gateway-routing_{context}"] += Gateway and routing troubleshooting + +[role="_abstract"] +When traffic is not flowing after you installed {mcpg}, you can investigate each component of the gateway routing to check system health. Depending on the errors you are receiving, you can troubleshoot at several layers. Breaks can occur at the gateway, route, or policy levels. + +If you have a `Connection Refused/Timeout` error and your client cannot reach the IP address, the cause might be the +listener. In this case, one of the following situations likely applies: + +* The port is not open. +* The load balancer has not assigned an IP address. +* The TLS handshake is failing. + +When you have this type of error, check the listener first. + +If can connect to the `Gateway `object, but you get an `HTTP` error, such as `404`, the cause can be a problem with the `HTTPRoute` custom resource (CR). The route exists, but the `Gateway` object has rejected it or the connection has failed. When you get these types of codes, check the `HTTPRoute` CR first. + +If requests either fail with a `503` error or bypass the router, this means that the route is recognized, but the connection to the backend failed or was not authorized properly. In this case, start with API-level checks and narrow your investigation to Envoy filters as needed. + +If the `EnvoyFilter` is not present, it usually means one of the following situations has occurred: + +* The `Gateway` CR status is not `Programmed`. +* There is a `labels` mismatch, and the MCP controller logic does not send any pods through the filter. +* The MCP controller is crashing or stuck. diff --git a/modules/proc-mcp-gateway-register-mcp-server.adoc b/modules/proc-mcp-gateway-register-mcp-server.adoc index 0823d94871da..128c709655d9 100644 --- a/modules/proc-mcp-gateway-register-mcp-server.adoc +++ b/modules/proc-mcp-gateway-register-mcp-server.adoc @@ -93,6 +93,11 @@ spec: * Replace the `spec.targetRef.namespace:` field value with the namespace where your `HTTPRoute` CR is applied. In this example, `__` is used. * Replace the `credentialRef.name:` field value with the name of your `Secret` CR. In this example, `__` is used. You can omit this parameter if your MCP server does not require authentication or authorization. * For more information about these parameters, see "Understanding the `MCPServerRegistration` custom resource." ++ +[IMPORTANT] +==== +A `toolPrefix` value cannot include spaces or special characters. +==== . Apply the CR by running the following command: + diff --git a/modules/proc-mcp-gateway-ts-gateway-listener-not-working.adoc b/modules/proc-mcp-gateway-ts-gateway-listener-not-working.adoc new file mode 100644 index 000000000000..e5a28c72aef1 --- /dev/null +++ b/modules/proc-mcp-gateway-ts-gateway-listener-not-working.adoc @@ -0,0 +1,84 @@ + +// Module included in the following assemblies: +// +// *mcp_gateway_config/mcp-gateway-troubleshooting.adoc + +:_mod-docs-content-type: PROCEDURE +[id="proc-mcp-gateway-ts-gateway-listener-not-working_{context}"] += Troubleshooting the gateway listener not working + +[role="_abstract"] +If your {mcpg} cannot reach an MCP endpoint at configured hostname, the cause might be that the `Listener` custom resource (CR) you configured is not working. You can troubleshoot this situation by using a few commands and some insight. + +Use the following concepts in conjunction with the commands that follow to solve a non-functioning `Listener` CR: + +* Ensure that your `Gateway` object has `Accepted` and `Programmed` conditions set to `True`. +* Verify that the `hostname` in the `Listener `CR matches your DNS or hosts configuration. + +.Prerequisites + +* You installed {mcpg}. +* You installed {prodname}. +* You configured a `Gateway` object. +* You configured an `HTTPRoute` object for the gateway. +* You installed the {oc-first}. +* You registered an MCP server. + +.Procedure + +. Check the general `Gateway` object configuration by running the following command: ++ +[source,terminal] +---- +$ oc get gateway -A +---- ++ +This command returns general information about all `Gateway` objects in the cluster. If the `Gateway` object you are troubleshooting does exist, the command returns the `gatewayClassName` is it using, whether or not it has an IP address or hostname assigned, and a `status`, such as `Ready`, `Programmed`, or `Pending`. + +. Check the full metadata and status history for one specific `Gateway` object by running the following command: ++ +[source,terminal,subs="+quotes"] +---- +$ oc describe gateway __ -n __ +---- ++ +* Replace `__` with the name of the `Gateway` object. +* Replace `__` with the namespace where the `Gateway` object is applied. +* This command can help you figure out why a `Gateway` object is stuck in `Pending` by checking for port conflicts and verifying that `SSL/TLS` certificates are correctly attached to `Listener` CRs. + +. Verify the `Listener` CR configuration by running the following command: ++ +[source,terminal,subs="+quotes"] +---- +$ oc get gateway __ -n __ -o yaml | grep -A 10 listeners +---- ++ +* Replace `__` with the name of the `Gateway` object. +* Replace `__` with the namespace where the `Gateway object` is applied. + +. Check all of your `Listener` CR configurations at the same time by running the following command: ++ +[source,terminal,subs="+quotes"] +---- +$ oc get gateway __ -n __ -o jsonpath='{range .spec.listeners[*]}{.name}{"\t"}{.hostname}{"\t"}{.port}{"\n"}{end}' | jq +---- +* Replace `__` with the name of the `Gateway` object. +* Replace `__` with the namespace where your `Gateway` object is applied. +* This command uses `jq` for formatting. + +. Check that the Istio gateway pod is running by using the following command: ++ +[source,terminal,subs="+quotes"] +---- +$ oc get pods -n __ -l istio=ingressgateway +---- ++ +* Replace `__` with the name of your `Gateway` object deployment. +* This command checks the status of Envoy-proxy pods and returns pod, traffic flow, and policy errors. + +. Verify that the port you are trying to use is not already in use by running the following command: ++ +[source,terminal] +---- +$ oc get gateway -A -o yaml | grep "port:" +---- diff --git a/modules/proc-mcp-gateway-ts-pods-not-starting.adoc b/modules/proc-mcp-gateway-ts-pods-not-starting.adoc new file mode 100644 index 000000000000..5f2b55b7cc10 --- /dev/null +++ b/modules/proc-mcp-gateway-ts-pods-not-starting.adoc @@ -0,0 +1,57 @@ +// Module included in the following assemblies: +// +// *mcp_gateway_config/mcp-gateway-troubleshooting.adoc + +:_mod-docs-content-type: PROCEDURE +[id="proc-mcp-gateway-ts-pods-not-starting_{context}"] += {mcpg} pods not starting + +[role="_abstract"] +After installation, if your {mcpg} pods are stuck in one of several states that indicate that they are not starting as expected, you can take several steps to diagnose the problem. + +Common causes include the following states and indicate an associated action: + +* `ImagePullBackOff`: Check image repository access and credentials. +* `CrashLoopBackOff`: Check the logs for application errors. +* `Pending:` Check resource availability and node capacity. +* `Init Container Failure*:` Check RBAC permissions. + +.Prerequisites + +* You installed {mcpg}. +* You installed {prodname}. +* You configured a `Gateway` object. +* You configured an `HTTPRoute` object for the gateway. +* You installed the {oc-first}. +* You registered an MCP server. + +.Procedure + +. Check the pod status by running the following command: ++ +[source,terminal,subs="+quotes"] +---- +$ oc get pods -n __ +---- ++ +Replace `__` with the name of the {mcpg} deployment that you are checking. + +. Describe problem pods by running the following command: ++ +[source,terminal,subs="+quotes"] +---- +$ oc describe pod -n __ __ +---- ++ +* Replace `__` with the name of the {mcpg} deployment that you are checking. +* Replace `__` with the name of the pod that you are checking. + +. Check the namespace logs by running the following command: ++ +[source,terminal,subs="+quotes"] +---- +$ oc logs -n __ __ +---- ++ +* Replace `__` with the name of the {mcpg} deployment that you are checking. +* Replace `__` with the name of the pod that you are checking. diff --git a/modules/proc-mcp-gateway-ts-requests-fail-or-bypass-router.adoc b/modules/proc-mcp-gateway-ts-requests-fail-or-bypass-router.adoc new file mode 100644 index 000000000000..983f3e77264d --- /dev/null +++ b/modules/proc-mcp-gateway-ts-requests-fail-or-bypass-router.adoc @@ -0,0 +1,152 @@ +// Module included in the following assemblies: +// +// *mcp_gateway_config/mcp-gateway-troubleshooting.adoc + +:_mod-docs-content-type: PROCEDURE +[id="proc-mcp-gateway-ts-requests-fail-or-bypass-router_{context}"] += Troubleshooting requests failing or bypassing the router + +[role="_abstract"] +When you are certain that an `MCPGatewayExtension` custom resource (CR) exists for your MCP server, but requests either fail or bypass the router, it might mean that the `EnvoyFilter` CR is not applied properly. You can take several steps to troubleshoot the problem. + +The `EnvoyFilter` CR is automatically created in the `Gateway` CR's namespace by the MCP gateway controller component when an `MCPGatewayExtension` CR is `Ready`. + +[IMPORTANT] +==== +You can look closely at the `EnvoyFilter` during deep troubleshooting, but do not manually edit or delete the CR. +==== + +.Prerequisites + +* You installed {mcpg}. +* You installed {prodname}. +* You configured a `Gateway` object. +* You configured an `HTTPRoute` object for the gateway. +* You installed the {oc-first}. +* You registered an MCP server. + +.Procedure + +. Ensure that the `Gateway` object exists and is in the expected namespace by checking the general `Gateway` object configuration by using the following command: ++ +[source,terminal] +---- +$ oc get gateway -A +---- ++ +This command returns general information about all `Gateway` objects in the cluster. If the `Gateway` object you are troubleshooting does exist, the command returns the `gatewayClassName` is it using, whether or not it has an IP address or hostname assigned, and a `status`, such as `Ready`, `Programmed`, or `Pending`. + +. Verify that the `MCPGatewayExtension` CR is `Ready` by running the following command: ++ +[source,terminal] +---- +$ oc get mcpgatewayextension -A` +---- + +. Verify that a `ReferenceGrant` CR exists if the `MCPGatewayExtension` CR is in a different namespace than the `Gateway` object by running the following command: ++ +[source,terminal,subs="+quotes"] +---- +$ oc get referencegrant __ -n __ -o yaml +---- ++ +* Replace `__` with the namespace where the `ReferenceGrant` CR is applied. +* Replace `__` with the names of the `ReferenceGrant` CR. + +. Check the `HTTPRoute` CR by running the following command: ++ +[source,terminal,subs="+quotes"] +---- +$ oc describe httproute __ -n __ +---- ++ +* Replace `__` with the namespace where the `HTTPRoute` CR is applied. +* Replace `__` with the names of the `HTTPRoute` CR. +* If the `HTTPRoute` CR is not `Accepted` by the `Gateway` object, the route is not programmed into Envoy, causing a `404`. +* The conditions `Status.Parents.Conditions: Accepted: True` and `Programmed: True` show that the route is correct. + +. If any of the CRs you just checked are not `Ready`, check the controller logs for `EnvoyFilter` creation errors by running the following command: ++ +[source,terminal,subs="+quotes"] +---- +$ oc logs -n __ deployment/mcp-gateway-controller +---- ++ +* Replace `__` with the name of your MCP gateway deployment. +* This step verifies that the MCP controller is successfully generating the underlying Istio configurations. + +. Check that the `EnvoyFilter` exists in the `Gateway` namespace by running the following command: ++ +[source,terminal,subs="+quotes"] +---- +$ oc get envoyfilter -n __ -l app.kubernetes.io/managed-by=mcp-gateway-controller +---- ++ +* Replace `__` with the namespace where the `Gateway` object is applied. + +. Verify the `EnvoyFilter` configuration by running the following command: ++ +[source,terminal,subs="+quotes"] +---- +$ oc describe envoyfilter -n __ -l app.kubernetes.io/managed-by=mcp-gateway-controller +---- ++ +* Replace `__` with the namespace where the `Gateway` object is applied. +* The `workloadSelector` labels must match your `Gateway` pods, or your policies are bypassed. + +. Compare the `EnvoyFilter` labels against your pod labels by running the following command: ++ +[source,terminal,subs="+quotes"] +---- +$ oc get pods -n __ --show-labels +---- ++ +* Replace `__` with the namespace where the `Gateway` object is applied. + +. Identify the port that the `Gateway` object is configured to use by running the following command: ++ +[source,terminal,subs="+quotes"] +---- +$ oc get gateway -n __ -o jsonpath='{range .spec.listeners[*]}{.name}{": "}{.port}{"\n"}{end}' +---- ++ +* Replace `__` with the name of the `Gateway` object. +* Replace `__` with the namespace where the `Gateway` object is applied. + +. Confirm that Envoy opened the `Gateway`-specified port by running the following command: ++ +[source,terminal,subs="+quotes"] +---- +$ oc exec -n deploy/-istio -- curl -s localhost:15000/listeners | grep -E ":[0-9]+" +---- + +. Verify the `EnvoyFilter` chain binding by running the following command: ++ +[source,terminal,subs="+quotes"] +---- +$ oc exec -n __ deploy/__-istio -- curl -s localhost:15000/config_dump | jq '.configs[] | select(.["@type"] | contains("ListenersConfigDump")) | .dynamic_listeners[] | select(.name | contains(":"))' +---- ++ +* Replace `__` with the name of the `Gateway` object. +* Replace `__` with the namespace where the `Gateway` object is applied. +* Check for `envoy.filters.http.ext_authz` for Authorino or `envoy.filters.http.ratelimit` for Limitador within the specific port config. + +. Check the Istio gateway pod configuration by running the following command: ++ +[source,terminal,subs="+quotes"] +---- +$ oc exec -n __ deploy/__-istio -- curl localhost:15000/config_dump | grep ext_proc +---- ++ +* Replace `__` with the namespace where the `Gateway` object is applied. +* Replace `__` with the names of the `Gateway` object. +* If this command returns empty, the `EnvoyFilter` CR is not active on this pod. Traffic is bypassing your policies. + +. Restart the Istio gateway to force a configuration reload by running the following command: ++ +[source,terminal,subs="+quotes"] +---- +$ oc rollout restart deployment/__-istio -n __` +---- +* Replace `__` with the namespace where the `Gateway` object is applied. +* Replace `__` with the names of the `Gateway` object. diff --git a/modules/proc-mcp-gateway-ts-traffic-not-reaching-backend-server.adoc b/modules/proc-mcp-gateway-ts-traffic-not-reaching-backend-server.adoc new file mode 100644 index 000000000000..5778ab74c146 --- /dev/null +++ b/modules/proc-mcp-gateway-ts-traffic-not-reaching-backend-server.adoc @@ -0,0 +1,67 @@ +// Module included in the following assemblies: +// +// *mcp_gateway_config/mcp-gateway-troubleshooting.adoc + +:_mod-docs-content-type: PROCEDURE +[id="proc-mcp-gateway-ts-traffic-not-reaching-backend-server_{context}"] += Troubleshooting traffic not reaching the backend MCP server + +[role="_abstract"] +When you are certain that an `HTTPRoute` custom resource (CR) exists for your application, but traffic is not reaching your backend MCP servers, you can take several steps to troubleshoot the problem. + +On the client side, errors such as `401`, `403`, and `404` can indicate this situation. + +.Prerequisites + +* You installed {mcpg}. +* You installed {prodname}. +* You configured a `Gateway` object. +* You configured an `HTTPRoute` object for the gateway. +* You installed the {oc-first}. +* You registered an MCP server. + +.Procedure + +. Check the `HTTPRoute` general custom resource (CR) status by running the following command: ++ +[source,terminal] +---- +$ oc get httproute -A +---- ++ +* This command returns general information about all `HTTPRoute` objects in the cluster. +* Check for the `Accepted` condition in the `HTTPRoute` CR `status` fields. + +. Check the full metadata and status history for one specific `HTTPRoute` object by running the following command: ++ +[source,terminal,subs="+quotes"] +---- +$ oc describe httproute __ -n __ +---- ++ +* Replace `__` with the name of the `HTTPRoute` object. +* Replace `__` with the namespace where the `HTTPRoute` object is applied. +* Verify that the `hostnames` value in the `HTTPRoute` CR matches the gateway `Listener` CR `hostname`. +* If the `HTTPRoute` status shows `Accepted: False`, then the `Gateway` object is not using the route. +* If the condition is `ResolvedRefs: False:`, the route is accepted through the `Gateway` object, but it cannot find the backend MCP service. There might be either a mismatch in the CR `metadata.name:` field, or the MCP service is in a namespace the `Gateway` object cannot access. + +. Verify the parent reference by running the following command: ++ +[source,terminal,subs="+quotes"] +---- +$ oc get httproute __ -n __ -o yaml | grep -A 5 parentRefs +---- ++ +* Replace `__` with the name of the `HTTPRoute` object. +* Replace `__` with the namespace where the `HTTPRoute` object is applied. +* Ensure that the retrieved `parentRefs` value matches your `Gateway` CR name and namespace exactly. + +. Check that the `allowedRoutes.namespaces` value in the `Gateway` CR allows the `HTTPRoute` namespace by running the following command: ++ +[source,terminal,subs="+quotes"] +---- +$ oc get gateway __ -n __ -o jsonpath='{range .spec.listeners[*]}{.name}{": "}{.allowedRoutes.namespaces.from}{"\n"}{end}' +---- ++ +* Replace `__` with the name of the `Gateway` object. +* Replace `__` with the namespace where the `Gateway` object is applied. diff --git a/modules/proc-register-ext-mcp-server-mcpserverregistration.adoc b/modules/proc-register-ext-mcp-server-mcpserverregistration.adoc index 3009228525f0..1d8269293e4b 100644 --- a/modules/proc-register-ext-mcp-server-mcpserverregistration.adoc +++ b/modules/proc-register-ext-mcp-server-mcpserverregistration.adoc @@ -48,6 +48,11 @@ spec: * Replace the `spec.targetRef.name:` field value with the name of the `HTTPRoute` CR you applied. * Replace the value of `spec.targetRef.namespace:` with the namespace where your `HTTPRoute` CR is applied. * The `spec.credentialRef:` field points to the `Secret` CR that has credentials for the external MCP server. ++ +[IMPORTANT] +==== +A `toolPrefix` value cannot include spaces or special characters. +==== . Apply the CR by running the following command: + diff --git a/observe_troubleshoot/mcp-gateway-troubleshooting.adoc b/observe_troubleshoot/mcp-gateway-troubleshooting.adoc index 896a481b134c..d565e07d8612 100644 --- a/observe_troubleshoot/mcp-gateway-troubleshooting.adoc +++ b/observe_troubleshoot/mcp-gateway-troubleshooting.adoc @@ -5,4 +5,15 @@ include::_attributes/attributes.adoc[] :context: mcp-gateway-troubleshooting [role="_abstract"] -FPO assembly +You can troubleshoot common issues with solutions when working with the MCP gateway across installation, configuration, and operation. + +//installation troubleshooting +include::modules/proc-mcp-gateway-ts-pods-not-starting.adoc[leveloffset=+1] + +include::modules/con-mcp-gateway-ts-gateway-routing.adoc[leveloffset=+1] + +include::modules/proc-mcp-gateway-ts-gateway-listener-not-working.adoc[leveloffset=+2] + +include::modules/proc-mcp-gateway-ts-traffic-not-reaching-backend-server.adoc[leveloffset=+2] + +include::modules/proc-mcp-gateway-ts-requests-fail-or-bypass-router.adoc[leveloffset=+2]