Is it a good idea to support predicting multiple instances upon one request?

[zyxue]

Currently, we're using the gPRC and predict_raw interface of seldon to do model serving. I wonder if there is any cons of supporting batch instances prediction (< 20 instances).

e.g. we are thinking of using a message like

message BatchRequest {
    repeated PredictionRequest requests = 1;
}

message PredictionRequest {
    feature_1 = 1;
    feature_2 = 2;
    ...
}

which is serialized to the binData field of SeldonMessage.

Similarly, the response would be

message BatchResponse {
    repeated PredictionResponse responses = 1;
}

Is it a good idea?

I wonder if supporting batch prediction would make it impossible to integrate with more complex features of Seldon in the future like the those in the "Complex Seldon Core Inference Graph" as shown in this picture

[adriangonz]

Hey @zyxue, I'm not sure if you've checked out yet the early support for batch processing in Seldon Core? One of the points that it tries to tackle is precisely how to send / process large batches of requests.

It would be good to hear your thoughts on that!

[zyxue]

the batch processing in Seldon Core is for offline use (not real time), right?

[ukclivecox]

You can use gRPC with a batch as long as any components in your graph can handle a batch of instances. As @adriangonz mentioned for offline tasks the batch processing route is probably more suited but it sounds like you are looking for real time.

[ukclivecox]

Closing now. Please reopen if further discussion needed