knative serving- Autoscaling- Grpc requests takes time to reach pod.

[msgurikar]

Hi,
We are using KNative serving version 1.11.1 with Isitio-Ingress gateway in our C# Grpc client service applications.
C# Grpc Service is deployed to AKS as KNative service with maxScale=80 and container concurrency=1.
We have C# grpc client, which will send 400 requests to kNative route address, Each request contains payload of 45MB data.
400 initial grpc requests some times seem to create 80 pods, but KNative doesnt send request to all 80 pods evenly, my initial 40 reqeusts reaches to the pod quickly, subsequent requests takes 1-5 minutes to reach the pod, due to this some pods unused are getting terminated by knative. I want to know how i can make use of all 80 pods to process my requests concurrently, so that my whole 400 requests should be in 5X minutes, if each request takes 1 minute to process.

Here is my Knative service yaml

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
name: test-service
namespace: testsvc
spec:
template:
metadata:
name: test-service-latest
annotations:
autoscaling.knative.dev/class: kpa.autoscaling.knative.dev
autoscaling.knative.dev/minScale: "1"
autoscaling.knative.dev/maxScale: "80"
autoscaling.knative.dev/scaleDownDelay: "30m"

spec:
  containerConcurrency: 1
  timeoutSeconds: 1000
  tolerations:
    - key: "sku"
      operator: "Equal"
      value: "qi"
      effect: "NoSchedule"
  nodeSelector:
    kubernetes.io/os: linux
    algo : linux-F16s_v2
  #timeoutSeconds: 600
  containers:       

  - image: sacr.azurecr.io/testservice_uni:latest
    imagePullPolicy: Always
    resources:
      requests:
        cpu: 950m
        memory: 950Mi
      limits:
        cpu: 1000m
        memory: 1000Mi

    ports:
      - name: h2c
        containerPort: 40083
    env:      
    
    #- name: POD_IP
    # valueFrom:
    #   fieldRef:
    #     fieldPath: status.podIP  

Thank you.

[msgurikar]

Hi,
Further to my question above, when i send large number of requests(around 80), i get the below error in activator logs

{"severity":"ERROR","timestamp":"2023-09-25T18:30:57.961415475Z","logger":"activator","caller":"network/error_handler.go:33","message":"error reverse proxying request; sockstat: sockets: used 90\nTCP: inuse 82 orphan 0 tw 12 alloc 119 mem 74\nUDP: inuse 0 mem 1\nUDPLITE: inuse 0\nRAW: inuse 0\nFRAG: inuse 0 memory 0\n","commit":"ff30afc","knative.dev/controller":"activator","knative.dev/pod":"activator-7fd96bc4c9-t8tp4","knative.dev/key":"testsvc/test-service-latest","error":"context canceled","stacktrace":"knative.dev/pkg/network.ErrorHandler.func1\n\tknative.dev/pkg@v0.0.0-20220118160532-77555ea48cd4/network/error_handler.go:33\nknative.dev/serving/pkg/activator/handler.(*activationHandler).proxyRequest.

Any suggestion to fix the problem,

Thank you.

[dprotaso]

This is probably related to this bug - #12387

Are you easily able to repro the issue?

[msgurikar]

Ok. Not consistently. Sometimes i get this.

Thank you.

[msgurikar]

Hi Dave,
Again i got this error, when i try to send large amount of data(90MB*80) to 80 pods at a same time. Any workaround for this error.

"severity":"ERROR","timestamp":"2023-09-26T18:44:07.974597732Z","logger":"activator","caller":"network/error_handler.go:33","message":"error reverse proxying request; sockstat: sockets: used 90\nTCP: inuse 82 orphan 0 tw 18 alloc 431 mem 124\nUDP: inuse 0 mem 1\nUDPLITE: inuse 0\nRAW: inuse 0\nFRAG: inuse 0 memory 0\n","commit":"f60eb32","knative.dev/controller":"activator","knative.dev/pod":"activator-54f7b7d8dd-k42lp","knative.dev/key":"testsvc/test-service-latest","error":"context canceled","stacktrace":"knative.dev/pkg/network.ErrorHandler.func1\n\tknative.dev/pkg@v0.0.0-20230718152110-aef227e72ead/network/error_handler.go:33\nknative.dev/serving/pkg/activator/handler.(*activationHandler).proxyRequest.func1\n\tknative.dev/serving/pkg/activator/handler/handler.go:140\nnet/http/httputil.(*ReverseProxy).ServeHTTP\n\tnet/http/httputil/reverseproxy.go:475\nknative.dev/serving/pkg/activator/handler.(*activationHandler).proxyRequest\n\tknative.dev/serving/pkg/activator/handler/handler.go:143\nknative.dev/serving/pkg/activator/handler.(*activationHandler).ServeHTTP.func1\n\tknative.dev/serving/pkg/activator/handler/handler.go:96\nknative.dev/serving/pkg/activator/net.(*revisionThrottler).try.func1\n\tknative.dev/serving/pkg/activator/net/throttler.go:236\nknative.dev/serving/pkg/queue.(*Breaker).Maybe\n\tknative.dev/serving/pkg/queue/breaker.go:155\nknative.dev/serving/pkg/activator/net.(*revisionThrottler).try\n\tknative.dev/serving/pkg/activator/net/throttler.go:226\nknative.dev/serving/pkg/activator/net.(*Throttler).Try\n\tknative.dev/serving/pkg/activator/net/throttler.go:521\nknative.dev/serving/pkg/activator/handler.(*activationHandler).ServeHTTP\n\tknative.dev/serving/pkg/activator/handler/handler.go:89\nknative.dev/serving/pkg/http/handler.(*timeoutHandler).ServeHTTP.func4\n\tknative.dev/serving/pkg/http/handler/timeout.go:118"}