Explore creation of a "batching" transformer

[Maximophone]

The unit would receive requests and aggregate them in a certain batch size.

The behaviour would be something like this:

Input transformation:
When receiving new request, store it in memory and wait (needs multithreading)
When enough requests have been received, returns a batch in one of them, nothing in the others

Output transformation:
Wait until all outputs have arrived
split the one containing the batch into all the others
return all

[erandagan]

It's worth mentioning that TFServing has a pretty flexible batching configuration
which can be used for inspiration.

[ukclivecox]

More likely to be part of core model servers such as tfserving or tensorrt inference server.