Skip to content

Conformance: Add Port Support to Echo Server #4229

@danehans

Description

@danehans

Currently the echo server provides pod name, namespace, etc in the response headers. For use cases such as running multiple echo servers in a single pod, the echo server should optionally support echo'ing the HTTP server listening port number in the response headers. For example:

# Data Parallelism (DP) backend deployment: 3 pods, each listening on three ports to simulate vLLM ranks
apiVersion: apps/v1
kind: Deployment
metadata:
  name: dp-inference-model-server-deployment
  namespace: inference-conformance-app-backend
  labels:
    app: dp-inference-model-server
spec:
  replicas: 3
  selector:
    matchLabels:
      app: dp-inference-model-server
  template:
    metadata:
      labels:
        app: dp-inference-model-server
    spec:
      containers:
      - name: echoserver-3000
        image: gcr.io/k8s-staging-gateway-api/echo-basic:v20240412-v1.0.0-394-g40c666fd
        ports:
        - containerPort: 3000
        readinessProbe:
          httpGet:
            path: /
            port: 3000
          initialDelaySeconds: 3
          periodSeconds: 5
          failureThreshold: 2
        env:
        - name: HTTP_PORT # Default port for HTTP echo server
          value: "3000"
        - name: H2C_PORT # Default port for HTC echo server
          value: "3001"
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
      - name: echoserver-3002
        image: gcr.io/k8s-staging-gateway-api/echo-basic:v20240412-v1.0.0-394-g40c666fd
        ports:
        - containerPort: 3002
        readinessProbe:
          httpGet:
            path: /
            port: 3002
          initialDelaySeconds: 3
          periodSeconds: 5
          failureThreshold: 2
        env:
        - name: HTTP_PORT
          value: "3002"
        - name: H2C_PORT
          value: "3003"
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
      - name: echoserver-3004
        image: gcr.io/k8s-staging-gateway-api/echo-basic:v20240412-v1.0.0-394-g40c666fd
        ports:
        - containerPort: 3004
        readinessProbe:
          httpGet:
            path: /
            port: 3004
          initialDelaySeconds: 3
          periodSeconds: 5
          failureThreshold: 2
        env:
        - name: HTTP_PORT
          value: "3004"
        - name: H2C_PORT
          value: "3005"
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP

xref GIE issue: kubernetes-sigs/gateway-api-inference-extension#1680

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions