cannot configure phoenix with envoy gateway in k8s?

[Dericko681]

I have deployed envoy gateway into a local Kubernetes cluster. I have tested with my open AI compatible model backend and it works, curl requests to the model return a 200 Ok response. I have installed and configured arize phoenix using the guide here https://aigateway.envoyproxy.io/docs/capabilities/observability/tracing but at the testing phase where i run kubectl logs -n envoy-ai-gateway-system deployment/phoenix | grep "POST /v1/traces" no traces are found. Have you done something similar to this pls share to step by step guide with me.
Please help me here
Thank you

[zirain]

cc @codefromthecrypt

[codefromthecrypt]

@Dericko681 steps here are for standalone mode which do not use k8s but might be helpful to verify your base state https://github.com/envoyproxy/ai-gateway/tree/main/cmd/aigw it is what the k8s directions you linked to were modeled after.

[Dericko681]

Hi @codefromthecrypt the commands in this doc https://aigateway.envoyproxy.io/docs/getting-started/prerequisites are for deploying envoy ai gateway using helm, from there up to the basic usage then skip to configure oberservability https://aigateway.envoyproxy.io/docs/capabilities/observability/tracing at the level the kubectl logs -n envoy-ai-gateway-system deployment/phoenix | grep "POST /v1/traces" no traces are returned but there seem to be a more complex issue

I still need more detailed guidance please

[codefromthecrypt]

to troubleshoot this with the information you've mentioned, just that there are no traces.. the next step would be for someone else to run through the same instructions and see if they also receive no traces. If they don't confirm what works or doesn't. like if the pod for pheonix is in error and that's why the grep fails or if there is an error in the ai gateway pod.

I am in UTC+8 and at the end of a day before some time off returning mid next week, I can run through this to verify if something in the instructions drifted when I'm back. If you need feedback before then someone else can do what I would do, which is what I just said above.

[benie-joy-possi]

Hi @codefromthecrypt,
I’m experiencing the same issue. I followed the commands in the docs for deploying Envoy AI Gateway using Helm and then configured observability as described. However, when I run:

kubectl logs -n envoy-ai-gateway-system deployment/phoenix | grep "POST /v1/traces"

no traces are returned, so it seems there might be a more complex issue at play. I’d appreciate any more detailed guidance you can provide.
Thanks!

Here are some screenshots for more clarity on the issue I am facing :

[Dericko681]

@codefromthecrypt all pods are running

I have tried using previous versions of the documentation right to 0.3 where observability was first introduced but the problem seems to have persisted form the beginning

[codefromthecrypt]

I ran through the whole guide start to finish, and there were a couple glitches:

Issue	Fix
OTEL endpoint missing namespace in helm value	Use phoenix-svc.envoy-ai-gateway-system:6006
Port forward service name	Use svc/phoenix-svc not svc/phoenix

Here are all the commands run, exactly, in my laptop which uses colima:

k8s-otel.md

Sorry about the trouble, I raised #1764 for the small fixes

[Dericko681]

oh I see the issue, the main issue here is the compatibility between ai gateway v0..0.0-latest with envoy gateway v0.0.0-latest. These two versions used together fails to export traces but using envoy gateway v1.6.0 and ai gateway v0.4.0 solves it all

thank you

[Dericko681]

use envoy gateway v1.6.0 and ai gateway v0.4.0