Is there a way to expose multiple ports on a single Knative service

[Tomcli]

Ask your question here:

Hi. We want to expose multiple ports on the Knative service. However, it seems like it doesn't allow more than one custom port according to the validation. https://github.com/knative/serving/blob/master/pkg/apis/serving/k8s_validation.go#L397-L406

Is there any way we could work around this? Thank you.

[vagababov]

Can you describe the use case?

[Tomcli]

We have a container that exposed http and grpc on two different port. we want to use a single Knative service to map those two ports like the following.

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: custom-sample-predictor
spec:
  template:
    metadata:
      annotations:
        autoscaling.knative.dev/class: kpa.autoscaling.knative.dev
        autoscaling.knative.dev/maxScale: "1"
        autoscaling.knative.dev/minScale: "1"
        autoscaling.knative.dev/target: "1"
        queue.sidecar.serving.knative.dev/resourcePercentage: "0.2"
    spec:
      containerConcurrency: 0
      containers:
      - env:
        - name: TARGET
          value: Python KFServing Sample
        image: tomcli/custom-sample
        ports:
          - name: h2c
            containerPort: 8080
          - name: http1
            containerPort: 8081
        name: user-container
        resources:
          limits:
            cpu: "2"
            memory: 4Gi
          requests:
            cpu: "2"
            memory: 2Gi
      timeoutSeconds: 60
  traffic:
  - latestRevision: true
    percent: 100

However, with the current validation test, we are only allowed to have one port per Knative service. Is there an other way that can achieve the same use case? Thanks

[vagababov]

/cc @mattmoor @evankanderson @dprotaso

[mattmoor]

This is complicated for a variety of reasons, and probably not something we'll support anytime soon (if ever). Most of the reasons that come to mind are networking related so cc @tcnghia

In the meantime, is it possible to multiplex the http handling alongside the GRPC handling?

[dprotaso]

@Tomcli can you comment on the specifics of your workload

I'm curious about the following:

1. What's the purpose for each port being exposed?
2. What's the exposure of each port (ie. are they both public to the internet)
3. What type of traffic/load does each port receive?
4. How do you expect this app to scale as the traffic/load increases/decreases.

[Tomcli]

Hi @dprotaso, one of our use cases is to use tensorflow serving on Knative.

Below are the answers to your questions:

1. TensorFlow serving has http and grpc hosted on two different port. They both are used for model prediction but with different protocol.
2. Both of the ports should be public to all users.
3. It depends on the model, most of the time the http port will receive restful requests such as Posting JSON formatted data. grpc port will receive data in TensorProto created by TensorFlow.
4. We are planning to scale based on the number of concurrent requests or cpu/gpu consumption.

[dprotaso]

We are planning to scale based on the number of concurrent requests or cpu/gpu consumption.

Is this scaling on the number of concurrent requests tied to a specific port?

[dprotaso]

TensorFlow serving has http and grpc hosted on two different port. They both are used for model prediction but with different protocol.

If it's a matter of a different protocols you could run two separate Knative Services.

[mattmoor]

Can GRPC not multiplex the GRPC handlers alongside non-GRPC http/2 handlers?

[dprotaso]

It requires some work and is language specific

We use it in our tests

serving/test/test_images/grpc-ping/main.go

Lines 80 to 88 in fd25321

	func httpWrapper(g *grpc.Server) http.Handler {
	return servingnetwork.NewProbeHandler(
	http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
	if r.ProtoMajor == 2 && r.Header.Get("Content-Type") == "application/grpc" {
	g.ServeHTTP(w, r)
	}
	}),
	)
	}

serving/vendor/knative.dev/pkg/network/h2c.go

Lines 29 to 36 in fd25321

	func NewServer(addr string, h http.Handler) *http.Server {
	h1s := &http.Server{
	Addr: addr,
	Handler: h2c.NewHandler(h, &http2.Server{}),
	}
	return h1s
	}

@Tomcli that's the other option worth exploring - then you'd have the expected behaviour you want with a single port

[Tomcli]

Thanks @dprotaso, we will try to explore these options and see can it fit in our scenario.