Table Of Contents
The multiscaleDeformableAttnPlugin is used to perform attention computation over a small set of key sampling points around a reference point rather than looking over all possible spatial locations. It makes use of multiscale feature maps to effectively represent objects at different scales. It helps to achieve faster convergence and better performance on small objects.
The multiscaleDeformableAttnPlugin takes 5 inputs in the following order : value, spatial_shapes, level_start_index, sampling_locations, and atttention_weights.
value
The input feature maps from different scales concatenated to provide the input feature vector. The shape of this tensor is [N, S, M, D] where N is batch size, S is the length of the feature maps, M is the number of attentions heads, D is hidden_dim/num_heads.
spatial_shapes
The shape of each feature map. The shape of this tensor is [L, 2] where L is the number of feature maps.
level_start_index
This tensor is used to find the sampling locations for different feature levels as the input feature tensors are flattened. The shape of this tensor is [L,].
sampling_locations
This tensor acts as a pre-filter for prominent key elements out of all the feature map pixels. The shape of this tensor is [N, Lq, M, L, P, 2] where P is the number of points, Lq is the length of feature maps(encoder)/length of queries(decoder).
attention_weights
This tensor consists of the attention weights whose shape is [N, Lq, M, L, P].
The multiscaleDeformableAttnPlugin generates the attention output of shape [N, Lq, M, D].
multiscaleDeformableAttnPlugin has plugin creator class multiscaleDeformableAttnPluginCreator and plugin class multiscaleDeformableAttnPlugin.
The plugin does not require any parameters to be built and used.
The following resources provide a deeper understanding of the multiscaleDeformableAttnPlugin plugin:
Networks:
For terms and conditions for use, reproduction, and distribution, see the TensorRT Software License Agreement documentation.
Feb 2022
This is the first release of this README.md file.
There are no known issues in this plugin.