Split OTLP endpoint path to fix Langfuse/LangSmith integration

[ChrisJBurns]

Summary

• The Go OTLP HTTP SDK's WithEndpoint() only accepts host:port, but platforms like Langfuse and LangSmith require a custom base path in the endpoint URL. When users configure an endpoint like https://cloud.langfuse.com/api/public/otel, the path component gets URL-encoded (slashes become %2F), breaking all requests to these backends.
• Parse the endpoint string into host:port and path components using a new splitEndpointPath helper, then pass the host to WithEndpoint() and the path (with signal suffix appended) to WithURLPath().

Fixes #4811

Type of change

• Bug fix

Test plan

• Unit tests (task test)
• Linting (task lint-fix)
• Manual testing (describe below)

Manual verification with Langfuse on Kind

Deployed a full Langfuse v3 stack (PostgreSQL, ClickHouse, Redis, MinIO) into a Kind cluster alongside the ToolHive operator built from this branch. Created an MCPTelemetryConfig with the exact scenario from the bug report — an endpoint with a custom path:

endpoint: "http://langfuse.langfuse.svc.cluster.local:3000/api/public/otel"

Before the fix, the proxy runner would log:

traces export: failed to send to http://langfuse...%2Fapi%2Fpublic%2Fotel/v1/traces

After the fix, traces export silently (no error), and OTLP trace JSON files appear in MinIO:

[2026-04-14 16:50:57 UTC] 1.9KiB otel/test-project/.../0ef0144d-...json
[2026-04-14 16:59:24 UTC] 1.9KiB otel/test-project/.../349cd6c0-...json

The trace payload contains correct span data with service.name: "echo-server", MCP-specific attributes (mcp.server.name, mcp.transport), and standard OTEL attributes.

Full reproduction steps

Prerequisites

• kind, kubectl, task, helm installed
• Clone this branch

1. Create cluster and deploy operator

task kind-setup-e2e
task operator-install-crds
task operator-deploy-local

2. Create namespaces

KUBECONFIG=kconfig.yaml kubectl create namespace langfuse
KUBECONFIG=kconfig.yaml kubectl create namespace mcp-test

3. Deploy Langfuse stack

Apply each manifest and wait for readiness:

KUBECONFIG=kconfig.yaml kubectl apply -f test/manual/langfuse/postgres.yaml
KUBECONFIG=kconfig.yaml kubectl wait --for=condition=ready pod -l app=postgres -n langfuse --timeout=120s

KUBECONFIG=kconfig.yaml kubectl apply -f test/manual/langfuse/redis.yaml
KUBECONFIG=kconfig.yaml kubectl wait --for=condition=ready pod -l app=redis -n langfuse --timeout=60s

KUBECONFIG=kconfig.yaml kubectl apply -f test/manual/langfuse/clickhouse.yaml
KUBECONFIG=kconfig.yaml kubectl wait --for=condition=ready pod -l app=clickhouse -n langfuse --timeout=120s

KUBECONFIG=kconfig.yaml kubectl apply -f test/manual/langfuse/minio.yaml
KUBECONFIG=kconfig.yaml kubectl wait --for=condition=ready pod -l app=minio -n langfuse --timeout=120s

KUBECONFIG=kconfig.yaml kubectl apply -f test/manual/langfuse/langfuse.yaml
KUBECONFIG=kconfig.yaml kubectl rollout status deployment/langfuse -n langfuse --timeout=300s

4. Deploy MCPServer with Langfuse telemetry

KUBECONFIG=kconfig.yaml kubectl apply -f test/manual/langfuse/mcp-with-langfuse.yaml
sleep 30
KUBECONFIG=kconfig.yaml kubectl get pods -n mcp-test

5. Send traffic and verify

# Send requests through the MCP proxy
for i in $(seq 1 5); do
  KUBECONFIG=kconfig.yaml kubectl run -n mcp-test "curl-$i" \
    --image=curlimages/curl --restart=Never --rm -it -- \
    curl -s -o /dev/null -w "%{http_code}\n" http://mcp-echo-server-proxy:8080/health
done

# Wait for OTLP batch flush and check proxy logs (no export errors = success)
sleep 15
PROXY=$(KUBECONFIG=kconfig.yaml kubectl get pods -n mcp-test -l app=mcpserver -o jsonpath='{.items[0].metadata.name}')
KUBECONFIG=kconfig.yaml kubectl logs -n mcp-test "$PROXY" -c toolhive --tail=5

# Verify traces landed in MinIO
KUBECONFIG=kconfig.yaml kubectl create -n langfuse -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: mc-check
spec:
  restartPolicy: Never
  containers:
  - name: mc
    image: minio/mc
    command: ["sh", "-c", "mc alias set m http://minio:9000 minioadmin minioadmin && mc ls --recursive m/langfuse/ && echo DONE"]
EOF

sleep 15
KUBECONFIG=kconfig.yaml kubectl logs -n langfuse mc-check
KUBECONFIG=kconfig.yaml kubectl delete pod mc-check -n langfuse

6. Teardown

task kind-destroy

Manifest: postgres.yaml

apiVersion: v1
kind: Service
metadata:
  name: postgres
  namespace: langfuse
spec:
  selector:
    app: postgres
  ports:
    - port: 5432
      targetPort: 5432
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: postgres
  namespace: langfuse
spec:
  replicas: 1
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
        - name: postgres
          image: postgres:16-alpine
          ports:
            - containerPort: 5432
          env:
            - name: POSTGRES_DB
              value: langfuse
            - name: POSTGRES_USER
              value: langfuse
            - name: POSTGRES_PASSWORD
              value: langfuse
          readinessProbe:
            exec:
              command: ["pg_isready", "-U", "langfuse"]
            initialDelaySeconds: 5
            periodSeconds: 5

Manifest: redis.yaml

apiVersion: v1
kind: Service
metadata:
  name: redis
  namespace: langfuse
spec:
  selector:
    app: redis
  ports:
    - port: 6379
      targetPort: 6379
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: redis
  namespace: langfuse
spec:
  replicas: 1
  selector:
    matchLabels:
      app: redis
  template:
    metadata:
      labels:
        app: redis
    spec:
      containers:
        - name: redis
          image: redis:7-alpine
          args: ["--maxmemory-policy", "noeviction"]
          ports:
            - containerPort: 6379
          readinessProbe:
            exec:
              command: ["redis-cli", "ping"]
            initialDelaySeconds: 5
            periodSeconds: 5

Manifest: clickhouse.yaml

apiVersion: v1
kind: Service
metadata:
  name: clickhouse
  namespace: langfuse
spec:
  selector:
    app: clickhouse
  ports:
    - name: http
      port: 8123
      targetPort: 8123
    - name: native
      port: 9000
      targetPort: 9000
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: clickhouse
  namespace: langfuse
spec:
  replicas: 1
  selector:
    matchLabels:
      app: clickhouse
  template:
    metadata:
      labels:
        app: clickhouse
    spec:
      containers:
        - name: clickhouse
          image: clickhouse/clickhouse-server:24-alpine
          ports:
            - containerPort: 8123
            - containerPort: 9000
          env:
            - name: CLICKHOUSE_DB
              value: default
            - name: CLICKHOUSE_USER
              value: default
            - name: CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT
              value: "1"
            - name: CLICKHOUSE_PASSWORD
              value: clickhouse
          readinessProbe:
            httpGet:
              path: /ping
              port: 8123
            initialDelaySeconds: 10
            periodSeconds: 5

Manifest: minio.yaml

apiVersion: v1
kind: Service
metadata:
  name: minio
  namespace: langfuse
spec:
  selector:
    app: minio
  ports:
    - name: api
      port: 9000
      targetPort: 9000
    - name: console
      port: 9001
      targetPort: 9001
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: minio
  namespace: langfuse
spec:
  replicas: 1
  selector:
    matchLabels:
      app: minio
  template:
    metadata:
      labels:
        app: minio
    spec:
      containers:
        - name: minio
          image: minio/minio:latest
          args: ["server", "/data", "--console-address", ":9001"]
          ports:
            - containerPort: 9000
            - containerPort: 9001
          env:
            - name: MINIO_ROOT_USER
              value: "minioadmin"
            - name: MINIO_ROOT_PASSWORD
              value: "minioadmin"
          readinessProbe:
            httpGet:
              path: /minio/health/ready
              port: 9000
            initialDelaySeconds: 5
            periodSeconds: 5
---
apiVersion: batch/v1
kind: Job
metadata:
  name: minio-create-bucket
  namespace: langfuse
spec:
  template:
    spec:
      restartPolicy: OnFailure
      containers:
        - name: mc
          image: minio/mc:latest
          command:
            - /bin/sh
            - -c
            - |
              sleep 5
              mc alias set myminio http://minio:9000 minioadmin minioadmin
              mc mb --ignore-existing myminio/langfuse
              echo "Bucket created"

Manifest: langfuse.yaml

apiVersion: v1
kind: Service
metadata:
  name: langfuse
  namespace: langfuse
spec:
  selector:
    app: langfuse
  ports:
    - name: http
      port: 3000
      targetPort: 3000
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: langfuse
  namespace: langfuse
spec:
  replicas: 1
  selector:
    matchLabels:
      app: langfuse
  template:
    metadata:
      labels:
        app: langfuse
    spec:
      containers:
        - name: langfuse
          image: langfuse/langfuse:3
          ports:
            - containerPort: 3000
          env:
            - name: DATABASE_URL
              value: "postgresql://langfuse:langfuse@postgres:5432/langfuse"
            - name: DIRECT_URL
              value: "postgresql://langfuse:langfuse@postgres:5432/langfuse"
            - name: CLICKHOUSE_CLUSTER_ENABLED
              value: "false"
            - name: CLICKHOUSE_URL
              value: "http://clickhouse:8123"
            - name: CLICKHOUSE_MIGRATION_URL
              value: "clickhouse://clickhouse:9000"
            - name: CLICKHOUSE_DB
              value: "default"
            - name: CLICKHOUSE_USER
              value: "default"
            - name: CLICKHOUSE_PASSWORD
              value: "clickhouse"
            - name: REDIS_HOST
              value: "redis"
            - name: REDIS_PORT
              value: "6379"
            - name: REDIS_AUTH
              value: ""
            - name: REDIS_CONNECTION_STRING
              value: "redis://redis:6379/0"
            - name: LANGFUSE_S3_EVENT_UPLOAD_BUCKET
              value: "langfuse"
            - name: LANGFUSE_S3_EVENT_UPLOAD_REGION
              value: "us-east-1"
            - name: LANGFUSE_S3_EVENT_UPLOAD_ACCESS_KEY_ID
              value: "minioadmin"
            - name: LANGFUSE_S3_EVENT_UPLOAD_SECRET_ACCESS_KEY
              value: "minioadmin"
            - name: LANGFUSE_S3_EVENT_UPLOAD_ENDPOINT
              value: "http://minio:9000"
            - name: LANGFUSE_S3_EVENT_UPLOAD_FORCE_PATH_STYLE
              value: "true"
            - name: LANGFUSE_S3_MEDIA_UPLOAD_BUCKET
              value: "langfuse"
            - name: LANGFUSE_S3_MEDIA_UPLOAD_REGION
              value: "us-east-1"
            - name: LANGFUSE_S3_MEDIA_UPLOAD_ACCESS_KEY_ID
              value: "minioadmin"
            - name: LANGFUSE_S3_MEDIA_UPLOAD_SECRET_ACCESS_KEY
              value: "minioadmin"
            - name: LANGFUSE_S3_MEDIA_UPLOAD_ENDPOINT
              value: "http://minio:9000"
            - name: LANGFUSE_S3_MEDIA_UPLOAD_FORCE_PATH_STYLE
              value: "true"
            - name: NEXTAUTH_URL
              value: "http://langfuse.langfuse.svc.cluster.local:3000"
            - name: NEXTAUTH_SECRET
              value: "test-secret-for-kind-cluster-only"
            - name: SALT
              value: "test-salt-for-kind-cluster-only"
            - name: ENCRYPTION_KEY
              value: "0000000000000000000000000000000000000000000000000000000000000000"
            - name: LANGFUSE_INIT_ORG_ID
              value: "test-org"
            - name: LANGFUSE_INIT_ORG_NAME
              value: "Test Org"
            - name: LANGFUSE_INIT_PROJECT_ID
              value: "test-project"
            - name: LANGFUSE_INIT_PROJECT_NAME
              value: "Test Project"
            - name: LANGFUSE_INIT_PROJECT_PUBLIC_KEY
              value: "pk-lf-test-public-key"
            - name: LANGFUSE_INIT_PROJECT_SECRET_KEY
              value: "sk-lf-test-secret-key"
            - name: LANGFUSE_INIT_USER_EMAIL
              value: "admin@test.local"
            - name: LANGFUSE_INIT_USER_NAME
              value: "Admin"
            - name: LANGFUSE_INIT_USER_PASSWORD
              value: "password"
          readinessProbe:
            httpGet:
              path: /api/public/health
              port: 3000
            initialDelaySeconds: 60
            periodSeconds: 10
            timeoutSeconds: 5
            failureThreshold: 12
          resources:
            requests:
              memory: "512Mi"
              cpu: "250m"
            limits:
              memory: "1Gi"

Manifest: mcp-with-langfuse.yaml

---
apiVersion: toolhive.stacklok.dev/v1alpha1
kind: MCPTelemetryConfig
metadata:
  name: langfuse-otel
  namespace: mcp-test
spec:
  openTelemetry:
    enabled: true
    # This is the key: endpoint WITH a path — the bug in #4811
    endpoint: "http://langfuse.langfuse.svc.cluster.local:3000/api/public/otel"
    insecure: true
    tracing:
      enabled: true
      samplingRate: "1.0"
    headers:
      Authorization: "Basic cGstbGYtdGVzdC1wdWJsaWMta2V5OnNrLWxmLXRlc3Qtc2VjcmV0LWtleQ=="
---
apiVersion: toolhive.stacklok.dev/v1alpha1
kind: MCPServer
metadata:
  name: echo-server
  namespace: mcp-test
spec:
  image: "ghcr.io/stackloklabs/yardstick/yardstick-server:1.1.1"
  transport: sse
  telemetryConfigRef:
    name: langfuse-otel

Changes

File	Change
pkg/telemetry/providers/otlp/endpoint.go	New splitEndpointPath helper that separates host:port from path
pkg/telemetry/providers/otlp/endpoint_test.go	Table-driven tests for the helper (Langfuse, LangSmith, trailing slash, etc.)
pkg/telemetry/providers/otlp/tracing.go	Use splitEndpointPath, add WithURLPath(basePath+"/v1/traces")
pkg/telemetry/providers/otlp/metrics.go	Use splitEndpointPath, add WithURLPath(basePath+"/v1/metrics")
pkg/telemetry/providers/otlp/tracing_test.go	Add test case for endpoint with custom path
pkg/telemetry/providers/otlp/metrics_test.go	Add test cases for endpoint with custom path

Does this introduce a user-facing change?

Yes. Users can now configure OTLP endpoints with custom base paths (e.g., https://cloud.langfuse.com/api/public/otel) in both MCPTelemetryConfig CRDs and inline telemetry config. Previously, any path in the endpoint was URL-encoded, breaking integration with Langfuse, LangSmith, and collectors behind reverse proxies with path prefixes.

Implementation plan

Approved implementation plan

1. Add splitEndpointPath helper in pkg/telemetry/providers/otlp/endpoint.go that splits a scheme-stripped endpoint into host:port and base path
2. In tracing.go, call splitEndpointPath and conditionally add WithURLPath(basePath+"/v1/traces")
3. In metrics.go, same treatment with WithURLPath(basePath+"/v1/metrics")
4. Add comprehensive tests for the helper and new exporter behavior

Key SDK details: WithURLPath() replaces the entire URL path (does not append to default), so we must manually construct basePath + "/v1/traces" and basePath + "/v1/metrics". The SDK's cleanPath() normalizes the path afterward.

Generated with Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 69.12%. Comparing base (bb213f1) to head (7db23ea).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #4815   +/-   ##
=======================================
  Coverage   69.12%   69.12%           
=======================================
  Files         530      531    +1     
  Lines       55157    55170   +13     
=======================================
+ Hits        38125    38136   +11     
- Misses      14105    14106    +1     
- Partials     2927     2928    +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[ChrisJBurns]

Multi-Agent Consensus Review

Agents consulted: site-reliability-engineer, go-expert-developer, code-reviewer, toolhive-expert

Consensus Summary

#	Finding	Consensus	Severity	Action
1	splitEndpointPath not defensive against scheme-prefixed endpoints	7/10	MEDIUM	Discuss
2	Missing test case for bare trailing slash endpoint	9/10	LOW	Fix
3	Hardcoded signal path suffixes	10/10	LOW	Fix
4	Removed error tests reduce negative coverage	10/10	MEDIUM	Fix

Overall

This is a well-crafted, focused bug fix with thorough manual verification against Langfuse on Kind. The approach is sound: splitEndpointPath correctly handles the OTLP HTTP SDK's expectation of host:port in WithEndpoint() while preserving custom URL paths via WithURLPath(). Backward compatibility is maintained — existing setups without custom paths are completely unaffected because WithURLPath is only called when basePath != "".

All four agents unanimously agree the core approach is correct. The findings are primarily about test hardening: restoring error-path coverage that was removed (F4, unanimous), adding an edge-case test for bare trailing slashes (F2), and a minor code hygiene suggestion to extract hardcoded OTLP path constants (F3). The one MEDIUM finding about scheme handling (F1) is a pre-existing issue in the CLI path that predates this PR — it's worth a follow-up but should not block merging.

---

Generated with Claude Code