Dynamic input shape?

[the-final]

Does dynamic input shape support?
Whether reoptimization is required after modifying the input shape

[diegofiori]

Hi @the-final!

Yes dynamic shape is supported by nebullvm. It is a built-in feature of the huggingface optimization, and it can be disabled passing the parameter use_static_shape=True to the optimize_huggingface_model function. For optimization using directly the pytorch and tensorflow APIs you need to directly pass an argument keyword named dynamic_axis. I attach here the description we gave of the argument in the docstrings:

dynamic_axis (Dict, optional): Dictionary containing info about the
            dynamic axis. It should contain as keys both "inputs" and "outputs"
            and as values two lists of dictionaries where each dictionary
            represents the dynamic axis information for an input/output tensor.
            The inner dictionary should have as key an integer, i.e. the
            dynamic axis (considering also the batch size) and as value a
            string giving a "tag" to it, e.g. "batch_size".

If there are some further constraints as maximum and minimum size of the input tensors along some dimension this information must be given using the keyword argument extra_input_info

extra_input_info (List[Dict], optional): List of extra information
            needed for defining the input tensors, e.g. max_value and min_value
            the tensors can get.

Let's consider as example a computer vision model taking as input images which size can range from 256x256 to 512x512. In this case the two arguments should be: (we are supposing input_tensor_size = (batch_size, 3, -1, -1))

dynamic_axis = {"inputs": [{2: "dynamic_dim_x", 3: "dynamic_dim_y"}], "outputs" = [{}]}
extra_input_info = [{"min_sizes": (None, None, 256, 256)]
input_sizes = [(3, 512, 512)]

As you can see the maximum dimension should be given as actual input size and it will use as "preferred" size during optimization.

Note that in the min_sizes definition you should also consider the batch size, differently from the input_size one where you neglect the batch size (max) value since it is explicitly given in another function argument. And be aware that we just released support for dynamic input, performance varies greatly from case to case and is not always optimal. We are currently working on improving and simplifying the way nebullvm handles dynamic inputs.