Stable v0.3 merge master (#709)

* fix some failures in X86 UnitTest

* opt openvino binary layer impl

* add openvino argmax layer builder

* opt openvino unary layer builder impl

* [DOC] add MatConvertParam description

* [DOC] fix link error

* [CONVERTER2TNN][UPD] 1. default do not clean build directory; (#694)

Co-authored-by: tiankai <tiankai@tencent.com>

* [ARM][BUG] fix upsample cubic openmp too many args (#692)

* [ARM][BUG] fix upsample cubic openmp too many args

* [ARM] fix blob converter unit test on a32

* [ARM] modify arm82 compile option

Co-authored-by: seanxcwang <seanxcwang@tencent.com>
Co-authored-by: seanxcwang <66675860+seanxcwang@users.noreply.github.com>

* [CONVERTER2TNN][BUG]1.fix bug for convert2tnn build.sh (#700)

Co-authored-by: tiankai <tiankai@tencent.com>

* [TOOLS][BUG] fix model checker bug when only check output

* [ARM][BUG] fp16 reformat support multi-inputs

* [DOC] update model check doc and model_check cmd help info

* [TOOLS][CHG] fix model check info error and modify some return value

Co-authored-by: Dandiding <Dandiding@tencent.com>
Co-authored-by: neiltian <65950677+neiltian-tencent@users.noreply.github.com>
Co-authored-by: quinnrong94 <quinnrong@tencent.com>
Co-authored-by: lucasktian <lucasktian@tencent.com>
Co-authored-by: tiankai <tiankai@tencent.com>
Co-authored-by: quinnrong94 <67782915+quinnrong94@users.noreply.github.com>
Co-authored-by: seanxcwang <seanxcwang@tencent.com>
Co-authored-by: seanxcwang <66675860+seanxcwang@users.noreply.github.com>

CMakeLists.txt

@@ -221,6 +221,12 @@ if(TNN_SYMBOL_HIDE AND UNIX)
     set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -fvisibility=hidden -fvisibility-inlines-hidden")
 endif()
 
+if(TNN_X86_ENABLE)
+    add_subdirectory(source/tnn/device/x86)
+    set(TARGET_OBJECTS ${TARGET_OBJECTS} "$<TARGET_OBJECTS:TNNX86>")
+    set(TNN_CPU_ENABLE ON)
+endif()
+
 if(TNN_CPU_ENABLE)
     add_subdirectory(source/tnn/device/cpu)
     set(TARGET_OBJECTS ${TARGET_OBJECTS} "$<TARGET_OBJECTS:TNNCpu>")
@@ -234,11 +240,6 @@ if(TNN_ARM_ENABLE)
     endif()
 endif()
 
-if(TNN_X86_ENABLE)
-    add_subdirectory(source/tnn/device/x86)
-    set(TARGET_OBJECTS ${TARGET_OBJECTS} "$<TARGET_OBJECTS:TNNX86>")
-endif()
-
 if(TNN_OPENVINO_ENABLE)
     add_subdirectory(source/tnn/network/openvino)
     set(TARGET_OBJECTS ${TARGET_OBJECTS} "$<TARGET_OBJECTS:TNNOpenVINO>")

doc/cn/development/model_check.md

@@ -3,7 +3,7 @@
 [English Version](../../en/development/model_check_en.md)
 
 ## 一、工具的作用
-校验对应平台（OpenCL，Metal，Cuda，ARM）的模型输出结果是否正确。
+校验对应平台（OpenCL，Metal，Cuda，ARM，HuaweiNPU）的模型输出结果是否正确。
 
 ## 二、编译
 编译model_check工具需要将以下宏设置为ON：  
@@ -15,7 +15,7 @@
 ## 三、校验工具使用
 ### 1. 命令
 ```
-./model_check [-h] [-p] [-m] [-d] [-i] [-n] [-s] [-o] [-f] <param>
+./model_check [-h] [-p] <tnnproto> [-m] <tnnmodel> [-d] <device> [-i] <input> [-o] [-e] [-f] <refernece> [-n] <val> [-s] <val>
 ```
 ### 2. 参数说明
 
@@ -24,11 +24,12 @@
 |-h, --help         |        |       |输出命令提示。                                |  
 |-p, --proto        |&radic; |&radic;|指定tnnproto模型描述文件。                   |   
 |-m, --model        |&radic; |&radic;|指定tnnmodel模型参数文件。                   |  
-|-d, --device       |&radic; |&radic;|指定模型执行的平台，如OPENCL，ARM，METAL等。    |  
+|-d, --device       |&radic; |&radic;|指定模型执行的平台，如OPENCL，ARM，METAL，CUDA，HUAWEI_NPU等。    |  
 |-i, --input        |        |&radic;|指定输入文件。目前支持格式为：<br>&bull; 文本文件（文件后缀为.txt）<br>&bull; 常用图片格式文件（文件后缀为 .jpg .jpeg .png .bmp）<br>如果不指定，则会使用 (-1, 1) 随机输入|  
 |-n, --bias         |        |&radic;|预处理，仅对输入为图片时有效。对输入数据各通道进行bias操作，参数格式为：0.0,0.0,0.0|  
 |-s, --scale        |        |&radic;|预处理，仅对输入为图片时有效。对输入数据各通道进行scale操作，参数格式为：1.0,1.0,1.0|  
 |-o, --output       |        |       |是否保存最终的输出。                           |  
+|-e, --end          |        |       |仅校验模型的最终输出。                         |  
 |-f, --ref          |        |&radic;|采用指定输出进行结果对比。目前支持格式为：<br>&bull; 文本文件（文件后缀为.txt），数据存储按照NCHW格式，以换行符分隔。|  
 
 ## 四、执行脚本
@@ -38,7 +39,7 @@
 #### 1.2 执行脚本
 ```
 cd <path_to_tnn>/platforms/android/
-./model_check_android.sh -c -p
+./model_check_android.sh -c -m <tnnproto> -p
 ```
 ### 2. Linux
 #### 2.1. 编译脚本
@@ -53,4 +54,4 @@ cd <path_to_tnn>/platforms/linux/
 
 ## 五、工具限制
 * 目前只支持fp32的模型校验；
-* 目前只针对fp32精度下的结果进行校验；
+* 对于逐层校验，只针对fp32精度下的结果进行校验；对于最后结果校验，使用Auto精度进行校验。

doc/cn/user/api.md

@@ -78,15 +78,15 @@ config.cache_path = "";
     auto status = instance->SetInputMat(input_mat, input_cvt_param);
 ```
 
-TNN输入设定通过调用SetInputMat接口完成，需要传入的数据保存在input_mat中，input_cvt_param可设置scale和bias相关转换参数。
+TNN输入设定通过调用SetInputMat接口完成，需要传入的数据保存在input_mat中，input_cvt_param可设置scale和bias[相关转换参数](#MatConvertParam参数说明)。
 
 ### 步骤4. 输出获取
 
 ```cpp
     auto status = instance->GetOutputMat(output_mat);
 ```
 
-TNN输出获取通过调用GetOutputMat接口完成，输出结果将按照特定格式保存在output_mat中。
+TNN输出获取通过调用GetOutputMat接口完成，输出结果将按照特定格式保存在output_mat中。输出结果同样支持scale和bias[相关转换](#MatConvertParam参数说明)。
 
 ## 二、API详解
 
@@ -448,6 +448,16 @@ struct PUBLIC MatConvertParam {
 };
 ```
 
+#### MatConvertParam参数说明：  
+- `reverse_channel`: 默认为`false`，若需要交换图像的B和R维度，可将此参数设置为`true`。  
+    * 仅`N8UC3`和`N8UC4`类型的Mat支持reverse_channel，其他类型的Mat会忽略该参数。  
+    * `ConvertFromMat`和`ConvertToMat`过程都支持reverse_channel。  
+- `scale`和`bias`: scale默认为 `1`，bias默认为`0`，计算顺序为先乘scale，再加bias。  
+    * 所有类型的Mat都支持scale和bias。  
+    * `ConvertFromMat`和`ConvertToMat`过程都支持scale和bias。  
+    * 若指定的scale全为`1`，且bias全为`0`，或者使用默认的scale和bias值，则不做乘scale和加bias操作；否则用户需提供与channel大小对应的scale和bias值。  
+    * 对于多维数据，scale和bias中的数值顺序和推理过程使用的数据格式保持一致。例如，若模型实际使用BGR格式进行推理，则`ConvertFromMat`和`ConvertToMat`过程，无论reverse_channel与否，scale和bias都需按照BGR顺序指定。也可理解为，`ConvertFromMat`先reverse channel，再乘scale和加bias；`ConvertToMat`先乘scale和加bias，再reverse channel。  
+
 ### 9. utils/cpu\_utils.h
 提供CPU线程核绑定以及省电模式设定相关工具。
 

doc/en/development/model_check_en.md

@@ -3,7 +3,7 @@
 [中文版本](../../cn/development/model_check.md)
 
 ## I. Function
-Check whether the output of the model on corresponding platform (OpenCL, Metal, Cuda, ARM) is correct.
+Check whether the output of the model on corresponding platform (OpenCL, Metal, Cuda, ARM, HuaweiNPU) is correct.
 
 ## II. Compile
 To compile the model_check tool, the following macro must be set to ON:  
@@ -15,19 +15,20 @@ To compile the model_check tool, the following macro must be set to ON:
 ## III. Usage
 ### 1. Command
 ```
-./model_check [-h] [-p] [-m] [-d] [-i] [-n] [-s] [-o] [-f] <param>
+./model_check [-h] [-p] <tnnproto> [-m] <tnnmodel> [-d] <device> [-i] <input> [-o] [-e] [-f] <refernece> [-n] <val> [-s] <val>
 ```
 ### 2. Parameter Description
 |option           |mandatory|with value |description                                       |  
 |:------------------|:------:|:-----:|:-------------------------------------------|  
 |-h, --help         |        |       |Output command prompt.                                |  
 |-p, --proto        |&radic; |&radic;|Specify tnnproto model description file.                   |   
 |-m, --model        |&radic; |&radic;|Specify the tnnmodel model parameter file.                   |  
-|-d, --device       |&radic; |&radic;|Specify the platform on which the model is executed, such as OPENCL, ARM, METAL, etc.    |  
+|-d, --device       |&radic; |&radic;|Specify the platform on which the model is executed, such as OPENCL, ARM, METAL, CUDA, HUAWEI_NPU etc.    |  
 |-i, --input_path   |        |&radic;|Specify the input file. The currently supported formats are:<br>&bull; Text file (the file suffix is ​​.txt)<br>&bull; Common picture format files (file suffix is ​​.jpg .jpeg .png .bmp)<br>If not specified, (-1, 1) will be used for random input|  
 |-n, --mean         |        |&radic;|Pre-processing, mean operation on each channel of input data, parameter format: 0.0, 0.0, 0.0|  
 |-s, --scale        |        |&radic;|Pre-processing, scale the input data channels, the parameter format is: 1.0, 1.0, 1.0|  
 |-o, --output       |        |       |Whether to save the final output.                           |  
+|-e, --end          |        |       |Only check output of model.                           |  
 |-f, --ref          |        |&radic;|Use the specified output to compare the results. The currently supported formats are:<br>&bull; Text file (file suffix is ​​.txt), data storage is in NCHW format, separated by newline.|
 
 
@@ -39,7 +40,7 @@ Copy the tnnproto and tnnmodel files of the model to be verified into `<path_to_
 #### 1.2 Execute the script
 `` `
 cd <path_to_tnn>/platforms/android/
-./model_check_android.sh -c -p
+./model_check_android.sh -c -m <tnnproto> -p
 `` `
 ### 2. Linux
 #### 2.1. Compile the script
@@ -54,4 +55,4 @@ cd <path_to_tnn>/platforms/linux/
 
 ## V. Tool Restrictions
 * Currently the tool only supports fp32 model verification;
-* At present, only the fp32 results can be verified;
+* For per-layer model checking, only the fp32 precision is supported. For only-output model checking, the precision is decided by device automatically.

doc/en/user/api_en.md

@@ -83,15 +83,15 @@ config.network_type = TNN_NS::NETWORK_TYPE_HUAWEI_NPU;
     auto status = instance->SetInputMat(input_mat, input_cvt_param);
 ```
 
-TNN input is set by SetInputMat interface.The data to be passed in is saved to input_mat. Scale and bias could be configured in input_cvt_param.
+TNN input is set by SetInputMat interface.The data to be passed in is saved to input_mat. Scale and bias could be configured in input_cvt_param, following the [mat-convert-parameter description](#MatConvertParam-description).
 
 ### Step4. Output 
 
 ```cpp
     auto status = instance->GetOutputMat(output_mat);
 ```
 
-TNN output is exported by ObtainingGetOutputMat interface. The result would be saved to output_mat in a specific format.
+TNN output is exported by ObtainingGetOutputMat interface. The result would be saved to output_mat in a specific format. Scale and bias could also be configured following the [mat-convert-parameter description](#MatConvertParam-description).
 
 ## III. API Explanation
 
@@ -452,6 +452,16 @@ struct PUBLIC MatConvertParam {
 };
 ```
 
+#### MatConvertParam description:  
+- `reverse_channel`: The default is `false`. If blue channel and red channel of the input mat need to be reversed, the parameter could be set to `true`.  
+    * The parameter is only valid for mats of `N8UC3` or `N8UC4` type. For other types of mat, the parameter is ignored.  
+    * Both `ConvertFromMat` and `ConvertToMat` procedures support reverse of channel.  
+- `scale` and `bias`: The default of scale is `1`, and the default of bias is `0`. The input mat is first multiplied by the scale, and then added with the bias.  
+    * All types of mat support scale and bias.  
+    * Both `ConvertFromMat` and `ConvertToMat` procedures support scale and bias.  
+    * If scale values are all equal to `1`, and bias values are all equal to `0`, or the default scale and bias are used, the scale and bias procedure will be skipped. Otherwise, both numbers of scale values and bias values should be consistent with the input channel.  
+    * For multi-channel inputs, the order of scale values and bias values should be consistent with the data format used in model inference. For example, if the model actually uses BGR images to do inference, then scale and bias should follow BGR order for both `ConvertFromMat` and `ConvertToMat` procedures, no matter reverse channel or not. This is also equivalent to first reverse channel, then do scale and bias for `ConvertFromMat`, and first do scale and bias, then reverse channel for `ConvertToMat`.  
+
 ### 9. utils/cpu\_utils.h
 Provide tools that are related to CPU thread core binding and power saving mode setting.
 

source/tnn/device/x86/x86_blob_converter.cc

@@ -0,0 +1,30 @@
+// Tencent is pleased to support the open source community by making TNN available.
+//
+// Copyright (C) 2020 THL A29 Limited, a Tencent company. All rights reserved.
+//
+// Licensed under the BSD 3-Clause License (the "License"); you may not use this file except
+// in compliance with the License. You may obtain a copy of the License at
+//
+// https://opensource.org/licenses/BSD-3-Clause
+//
+// Unless required by applicable law or agreed to in writing, software distributed
+// under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR
+// CONDITIONS OF ANY KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations under the License.
+
+#include "tnn/core/macro.h"
+#include "tnn/utils/blob_converter_default.h"
+#include "tnn/utils/blob_converter.h"
+
+namespace TNN_NS {
+
+class X86BlobConverterAcc : public DefaultBlobConverterAcc {
+public:
+    X86BlobConverterAcc(Blob *blob) : DefaultBlobConverterAcc(blob) {}
+    ~X86BlobConverterAcc() {}
+};
+
+DECLARE_BLOB_CONVERTER_CREATER(X86);
+REGISTER_BLOB_CONVERTER(X86, DEVICE_X86);
+
+}  // namespace TNN_NS

source/tnn/network/openvino/layer_builder/abs_layer_builder.cc

@@ -20,42 +20,17 @@
 #include <ngraph/op/op.hpp>
 #include <ngraph/opsets/opset.hpp>
 #include <ngraph/opsets/opset1.hpp>
-#include <ngraph/opsets/opset2.hpp>
-#include <ngraph/opsets/opset3.hpp>
 #include <inference_engine.hpp>
 
 #include "tnn/layer/base_layer.h"
-#include "tnn/network/openvino/layer_builder/openvino_layer_builder.h"
-#include "tnn/extern_wrapper/foreign_blob.h"
-#include "tnn/extern_wrapper/foreign_tensor.h"
-#include "tnn/network/openvino/openvino_types.h"
-#include "tnn/network/openvino/custom_layer/custom_abs.h"
+#include "tnn/network/openvino/layer_builder/unary_layer_builder.h"
 
 namespace TNN_NS {
+namespace openvino {
 
-DECLARE_OPENVINO_LAYER_BUILDER(Abs, LAYER_ABS);
+DECLARE_UNARY_LAYER_BUILDER(Abs, LAYER_ABS);
 
-Status AbsOVLayerBuilder::Build() {
-
-    if (GetInputNodes().size() <=0) {
-        LOGE("Error: 0 input nodes\n");
-        return TNNERR_INIT_LAYER;
-    }
-    auto input_node = GetInputNodes()[0];
+REGISTER_UNARY_LAYER_BUILDER(Abs, LAYER_ABS);
 
-    auto absNode = std::make_shared<ngraph::op::Abs>(input_node->output(0));
-    // auto absNode = std::make_shared<CustomAbsOp>(input_node->outputs(), base_layer_, GetInputBlobs(), GetOutputBlobs());
-
-    absNode->set_friendly_name(param_->name);
-    absNode->validate_and_infer_types();
-
-    ngraph::NodeVector outputNodes;
-    outputNodes.push_back(absNode);
-    SetOutputTensors(outputNodes);
-
-    return TNN_OK;
 }
-
-REGISTER_OPENVINO_LAYER_BUILDER(Abs, LAYER_ABS);
-
 }

source/tnn/network/openvino/layer_builder/acos_layer_builder.cc

@@ -20,40 +20,17 @@
 #include <ngraph/op/op.hpp>
 #include <ngraph/opsets/opset.hpp>
 #include <ngraph/opsets/opset1.hpp>
-#include <ngraph/opsets/opset2.hpp>
-#include <ngraph/opsets/opset3.hpp>
 #include <inference_engine.hpp>
 
 #include "tnn/layer/base_layer.h"
-#include "tnn/network/openvino/layer_builder/openvino_layer_builder.h"
-#include "tnn/extern_wrapper/foreign_blob.h"
-#include "tnn/extern_wrapper/foreign_tensor.h"
-#include "tnn/network/openvino/openvino_types.h"
+#include "tnn/network/openvino/layer_builder/unary_layer_builder.h"
 
 namespace TNN_NS {
+namespace openvino {
 
-DECLARE_OPENVINO_LAYER_BUILDER(Acos, LAYER_ACOS);
+DECLARE_UNARY_LAYER_BUILDER(Acos, LAYER_ACOS);
 
-Status AcosOVLayerBuilder::Build() {
-
-    if (GetInputNodes().size() <=0) {
-        LOGE("Error: 0 input nodes\n");
-        return TNNERR_INIT_LAYER;
-    }
-    auto input_node = GetInputNodes()[0];
+REGISTER_UNARY_LAYER_BUILDER(Acos, LAYER_ACOS);
 
-    auto acosNode = std::make_shared<ngraph::op::Acos>(input_node->output(0));
-
-    acosNode->set_friendly_name(param_->name);
-    acosNode->validate_and_infer_types();
-
-    ngraph::NodeVector outputNodes;
-    outputNodes.push_back(acosNode);
-    SetOutputTensors(outputNodes);
-
-    return TNN_OK;
 }
-
-REGISTER_OPENVINO_LAYER_BUILDER(Acos, LAYER_ACOS);
-
 }

source/tnn/network/openvino/layer_builder/add_layer_builder.cc

@@ -23,53 +23,14 @@
 #include <inference_engine.hpp>
 
 #include "tnn/layer/base_layer.h"
-#include "tnn/network/openvino/layer_builder/openvino_layer_builder.h"
-#include "tnn/extern_wrapper/foreign_blob.h"
-#include "tnn/extern_wrapper/foreign_tensor.h"
-#include "tnn/network/openvino/openvino_types.h"
+#include "tnn/network/openvino/layer_builder/binary_layer_builder.h"
 
 namespace TNN_NS {
+namespace openvino {
 
-DECLARE_OPENVINO_LAYER_BUILDER(Add, LAYER_ADD);
+DECLARE_BINARY_LAYER_BUILDER(Add, LAYER_ADD);
 
-Status AddOVLayerBuilder::Build() {
+REGISTER_BINARY_LAYER_BUILDER(Add, LAYER_ADD);
 
-    if (GetInputNodes().size() <= 0) {
-        LOGE("Error: %d input nodes\n", GetInputNodes().size());
-        return TNNERR_INIT_LAYER;
-    }
-    auto input_node = GetInputNodes();
-
-    std::shared_ptr<ngraph::op::v1::Add> addNode;
-    if (input_node.size() == 2) {
-        addNode = std::make_shared<ngraph::op::v1::Add>(
-            input_node[0]->output(0), input_node[1]->output(0));
-    } else {
-        auto resource = dynamic_cast<EltwiseLayerResource*>(GetResource());
-        // suppose that weights are on channel broadcast
-        ngraph::Shape weightNodeShape;
-        for (size_t i = 0; i < input_node[0]->get_output_shape(0).size(); i++) {
-            if (i == 1) weightNodeShape.push_back(resource->element_handle.GetDataCount());
-            else weightNodeShape.push_back(1);
-        }
-
-        auto weightNode = std::make_shared<ngraph::op::Constant>(
-            ngraph::element::Type_t::f32, weightNodeShape, resource->element_handle.force_to<float*>());
-        weightNode->validate_and_infer_types();
-
-        addNode = std::make_shared<ngraph::op::v1::Add>(
-            input_node[0]->output(0), weightNode);
-    }
-    addNode->set_friendly_name(param_->name);
-    addNode->validate_and_infer_types();
-
-    ngraph::NodeVector outputNodes;
-    outputNodes.push_back(addNode);
-    SetOutputTensors(outputNodes);
-
-    return TNN_OK;
 }
-
-REGISTER_OPENVINO_LAYER_BUILDER(Add, LAYER_ADD);
-
 }