NNPACK UNIFIED API PROBLEM

[Darwin2011]

Hi,

I am trying to figure out NNPACK API usage.

• for convolution with stride = 1, I should use compute_convolution_output.
• for convolution with stride >= 2, I should use nnp_convolution_inference. In this case, patche2cols and sgemm will be used as computation backend.

Is there any possibility to integrate there two API into one and provide users one unified and transparent interface? That can make things much easy when using the library.

Best Regards.

[Maratyszcza]

The difference between inference and output (forward pass in training) functions is based on two considerations:

1. inference functions are optimized for batch = 1 case, while output functions are optimized for moderately large batch sizes (64 and above).
2. While none of the following is currently implemented, I have plans to merge some extra operations into convolutional/fully connected layers. The type of operations which can be merged, depends on whether the framework would need to run backward pass on the layer using the outputs:

• Both ReLU and ELU can be merged into the preceding convolutional (or fully connected) layer.
• If the model is used for inference, 2x2 pooling with 2x2 stride can be merged into preceding convolutional or fully connected layer (potentially with embedded ReLU/ELU). If the layer is used for training, this fusion is impossible, because the backward pass on the convolutional layer would need the output of the convolutional layer before 2x2 pooling, which wouldn't be produced by the forward pass.
• If the model is used for training, batch normalization layer can be partially merged into preceding convolutional or fully connected layer. Specifically, computation of per-channel mean values can be merged into the store stage of the preceding layer. In inference, batch normalization layer becomes a static scale + bias layer, and often can be statically merged into preceding convolutional or fully connected layer, so computation of per-channel mean activations makes no sense.

The reason why nnp_convolution_inference supports strides while nnp_convolution_output does not is that NNPACK is in active development and implicit_gemm algorithm and strided convolutions were implemented for inference function first. Support for implicit_gemm algorithm and strided convolutions in the training functions is in short-term plans, but for now you'd need to fall back to an implementation outside NNPACK.

One more clarification about implicit_gemm algorithm: it does not do patch2cols+sgemm, but rather does something smarter. High-performance implementations of SGEMM internally repack the matrix into cache-friendly form. Thus, a typical SGEMM-based implementation of a convolutional layer would involve two repacking operations: one inside patch2cols and another inside sgemm. NNPACK's implicit_gemm algorithm combines these two repacking operations into one; the main motivation is to operate with low memory overhead (NNPACK doesn't allocate memory for the whole patch2cols matrix, just for L3-sized block of it), but you'd likely find it performing better than traditional patch2cols+sgemm implementations due to fewer memory repacking operations.

[lironmo]

any update about the short term implementation, for stride size bigger than 1.