Paddle Tensor Operation Library initial implementation #34425

chenwhql · 2021-07-27T08:45:52Z

PR types

Breaking changes

PR changes

Others

Describe

一、背景

Paddle Tensor运算库（Operation Library）旨在解决目前paddle中OpKernel存在的诸多问题，包括但不限于：

内部二次研发效率：OpKernel开发可复用性较差，类似逻辑有多处重复代码
- 新增OpKernel需要去别的OpKernel中Copy代码
调度性能：动态图复用静态图Op体系，执行调度性能差，Kernel需要升级为函数式，支持动态图直观地调用
外部二次研发效率：paddle自定义算子开发复杂算子，基础逻辑需要自行实现，需要梳理形成C++ API体系，供用户复用内部计算实现
细粒度调度：OpKernel中糅合了框架调度逻辑，不够纯粹和具体，无法支持细粒度调度
- 例如if(use_mkldnn), if(use_cudnn)等调度属性的分支

二、目标

短期：
- OpKernel改造为函数式，支持内部研发复用，并能够兼容现有执行体系
- 运算类C++ API体系初步设计实现，支持在自定义算子中调用
长期：
- 通过框架不同方向复用同一 Tensor运算库，从根本上做到训练推理协同一致、基础组件稳定可靠、增量开发友好高效

三、设计

3.1 核心诉求

易用、可复用、开发效率高、理解成本低
独立、灵活、细粒度计算、可裁剪

3.2 Tensor设计

高层API Tensor：对用户可见的Tensor数据结构，与Python端概念一致
内部Tensor体系：异构Tensor体系设计，不同Tensor有不同的类

3.3 Kernel及API设计

API一致性：C++计算类API的函数参数列表与Python对应API一致，无模板，无Context参数
细粒度：Kernel函数尽可能解耦，每个函数的功能都尽可能简单和具体，Kernel函数没有默认及可选参数，如果需要，拆分为多个Kernel实现
Kernel与API一致性：函数式Kernel为了便于代码复用，可以有模板，Context类参数作为起始参数，随后的输入、输出、属性参数顺序与命名尽可能与API一致

3.4 命名、目录、命名空间

命名：pten, Paddle TENsor Operation Library
目录：

./paddle
    ./fluid (现有C++核心框架目录)
    ./pten (Tensor运算库)
        ./api (运算库low level API，可用于内部Kernel间开发复用)
        ./hapi (运算库high level API，面向外部用户使用、和Python API签名一致)
        ./common (运算库内部和外部hapi公用的基础组件)
        ./core (运算库基础组件，如底层Tensor及kernel注册管理组件)
        ./kernels (各设备的Kernels)
            ./cpu
            ./cuda
            ./cudnn
            ./metal
            ./mkldnn
            ...
            ./functions (由Kernel调用的底层的公共函数)
                ./eigen
                ./blas
                ...
        ./tests (运算库单测集合)
	...

命名空间
- 面向用户的hapi：调用路径、命名及参数与Python API严格一致
  - paddle.Tensor -> paddle::Tensor
  - paddle.nn.functional.relu -> paddle::nn::functional::relu
  - paddle.matmul -> paddle::matmul
- 面向Kernel开发者的api：采用单层命名空间，尽可能简洁
  - paddle::pten::Sign<T>
  - paddle::pten::Relu<T>
  - paddle::pten::Matmul<T>

3.5 编译

预计pten会作为底层计算库，独立编译，由上层框架执行体系调用，包括但不限于fluid，lite，cinnrt等，计算库在编译上不会反向依赖框架（现仍有依赖，后续会去掉）

四、现状说明

本PR只是初始实现，仅包含基础组件实现及少数简单Kernel的迁移验证，具体包括：

基础枚举数据结构datatype，datalayout，backend新增
Tensor及DenseTensor的基础实现
新Kernel的基础注册管理组件
共计6个kernel的简单迁移改写：sign，mean，scale，dot，fill_any_like，flatten
部分已迁移Kernel的API实现及测试
已迁移kernel在原有动静态图执行体系下的适配调用

TODO说明

以下问题希望暂时先忽略，初始PR并不是最终态，后续会跟进完善，目前初始PR需要尽快合入以便于后续并行推进

Place、DDim、DeviceContext暂时使用了原先fluid的实现，后续会重写计算库相关实现
SelectedRows会在后续PR补充实现，由于暂未支持SelectedRows，test_scale_op单测中对于rows的前后检查对比目前无法通过，暂时注释
sign和scale目前并未实现API，因为目前实现的API仅用于跑通验证，目的并不是全面覆盖，后续API会自动生成，现少数API实现的代码也会被移除
API单测中也只有CPU测试，仅出于跑通验证目的，后续API的测试会通过动态图调用，复用现有Python单测进行检查，少数API单测代码也会被移除
新kernel的测试是通过原Python端op单测检验的，执行调度上进行了适配，能够在原执行体系下直接调用新Kernel，并非没有单测
新增临时FLAG run_pten_kernel 用于初期过渡，并且暂时会默认为 false，目前已经设置 run_pten_kernel为true并通过全部CI，暂时改为 false 是为了不对现有模型的性能产生影响，目前直接调用新kernel会比调用原kernel略慢，后续会跟进优化
原则上改写一个kernel会将原先**_op.h中的OpKernel::Compute实现移除，但目前仍然保留了一层壳，是因为移除Kernel需要确保新Kernel适配所有执行体系，目前只适配的原动态图和静态图执行体系，新的执行器及预测Predictor暂未适配，后续会跟进适配并移除原Kernel实现
pten内部目录结构目前可能对于kernel代码复用还不够友好，在后续演变过程中还会有所调整，并非最终态
代码中还有诸多TODO，此处不一一展开了，后续都会跟进解决

paddle-bot-old · 2021-07-27T08:46:16Z

Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

… op2func_refactor

Shixiaowei02 · 2021-10-27T07:10:30Z

paddle/pten/hapi/lib/kernel_dispatch.h

+namespace experimental {
+
+// TODO(shixiaowei): replaced by new DeviceContext later
+using CPUContext = paddle::platform::CPUDeviceContext;


[TODO] 提供新的 Context 数据结构

Shixiaowei02 · 2021-10-27T07:12:33Z

paddle/pten/hapi/lib/utils/tensor_utils.cc

+  return {};
+}
+
+std::unique_ptr<pten::TensorBase> MakePtenTensorBaseFromVar(


[TODO] 这里的性能很关键，需要后续优化

Shixiaowei02 · 2021-10-27T07:13:20Z

paddle/fluid/framework/operator.cc

+    paddle::SmallVector<std::shared_ptr<pten::TensorBase>> tmp_outputs;
+    for (auto var : outs_vector) {
+      tmp_outputs.emplace_back(
+          experimental::MakePtenTensorBaseFromVar(var, out_def));


[TODO] 这里的性能很关键，后续需要优化

Shixiaowei02 · 2021-10-27T07:15:29Z

paddle/pten/core/utils/intrusive_ptr.h

+#include "glog/logging.h"
+#include "paddle/fluid/platform/enforce.h"
+
+namespace pten {


[TODO] 代码位置后续移动到 paddle/utils

Shixiaowei02 · 2021-10-27T07:17:19Z

paddle/pten/hapi/include/tensor.h

+ * another simple Tensor design may be required for inference.
+ */
+
+class Tensor final {


[TODO] 此高层 Tensor 是临时的，将以和 ext_tensor 的兼容作为第一目标

Shixiaowei02 · 2021-10-27T07:19:41Z

paddle/pten/core/allocator.h

+/// is to be compatible with allocators from different frameworks
+/// without significant performance loss. This class does not
+/// support being inherited.
+class Allocation final {


[TODO] 将双方框架的 Allocation 修改为此接口，减小不必要的开销

Shixiaowei02 · 2021-10-27T07:20:30Z

paddle/pten/hapi/lib/utils/storage.h

+  const int64_t size_{0};
+};
+
+class SharedStorage : public pten::Storage {


[TODO] 头文件和接口分离

Shixiaowei02 · 2021-10-27T07:21:30Z

paddle/pten/tests/CMakeLists.txt

@@ -0,0 +1,10 @@
+cc_test(pten_backend_test SRCS backend_test.cc DEPS gtest)


[TODO] tests/ 目录结构需要细化，明确依赖

Shixiaowei02 · 2021-10-27T07:23:20Z

paddle/pten/kernels/cpu/CMakeLists.txt

@@ -0,0 +1,5 @@
+cc_library(math_cpu SRCS math.cc DEPS dense_tensor kernel_context kernel_factory eigen_function)


[TODO] 后续另行提交 PR，目录结构建议细化为 x86 和 host，不用 cpu 泛指

jeff41404

lgtm

Superjomn

LGTM

先有一个base

Avin0323

LGTM for PR-CI-OP-benchmark

Superjomn

LGTM

Avin0323

LGTM for unity_build_rule.cmake

…34425) * initial tensor design & sign kernel demo * add move constructor for meta & add lodtensor * add dirs & sign xpu kernel * add mean cpu&cuda kernel impl * move sign & mean xpu & npu kernel * add selected_rows basic impl * refactor design, BaseTensor to DenseTensor, etc. * add scale mkldnn kernel * polish xpu & npu impl details * fix mkldnn reuse compile failed * change tensor operation lib name * rename util filename * add more comments * change TensorImplInterface to TensorInterface * add kernel key and factory * remove MKLDNNTensorMeta, add MKLDNNDenseTensor * change XXDeviceContext to XXContext * add base kernel registrar utils & test on sign * replace boost::any by paddle::any * fix several ci failed * fix npu compile error * add ordered map util * fix multiple ordered_map compile errors * move dev into include dir * support sign op in static op run * fix static op run error * fix new executor compile failed * add dygraph branch & remove sign_op.h * fix test_infer_no_need_buffer_slots * fix rocm compile link error * fix unitybuild error & clear glog * fix npu compile failed * skip quant trans test * fix part windows compile problem * fix xpu enforce error * fix inference test failed * remove ordered_map to solve quant failed * fix part of rcom compile faild * add more register kernels * revert scale kernel temporarily * fix code format error * add new kernel registrar marco * rename top to tcmpt * revert xpu, npu, mkldnn impl & remove op def * add kernel args parse functor to auto parse args * revert some change & add scale kernels * add op proto in dygraph kernelcontext building * polish kernel dispatch logic & nameing rule * fix scale kernel match error * fix scale test failed * add mean API and unittest * test mean api success * add branch to solve compiled error * skip clang format error * add mean skip rule in op_library * add dot kernel, api and unittest (PaddlePaddle#6) * remove old kernel and add symbol link * fix dot compiled failed * add merco for module declare * fix npu and xpu compile error * revert sign, mean, scale, dot kernel removing * add comment for keeping old kernel impl * fix mutable_data error * fix bfloat16 conflit * fix inference undef error * adapt to msvc compile rules * polish comment for template inst * add cmake template instantiation for win * fix backend to place device id bug * fix ifdef error * Op2functor (PaddlePaddle#7) * add kernel args maker class * make args maker non-const * remove debug log * modify codes by review options * split constructPrKernelContext function * fix output name bug * fix test_mean_op test_sign_op failed * fill_any_like kernel refactor (PaddlePaddle#10) * fill_any_like kernel refactor * remove useless code of full_like c++ api * skip dtype for fill_any_like * add attrs for kernel key constrcut * add use_pt_kernel Flags to control whether to use pt kernel (PaddlePaddle#13) * add use_pt_kernel Flags to control whether to use pt kernel * change the default value to true for cheking pt kernels * fix mutable_data cuda place error * move high level apis into hapi * remove selectedrows adapting temporarily * Support Scalar in Tensor Compute Library (PaddlePaddle#14) * fill_any_like kernel refactor * remove useless code of full_like c++ api * Support Scalar in Tensor Compute Library * add scalar in dygraph and static graph mode * keep the basic type for attr, instead of using scalar for all * merge the code * remove mkldnn tensor & polish details * use flat_hash_map and small_vector in kernel factory * Refactor flatten kernel (PaddlePaddle#12) * refactor flatten kernel * update infershape function * fix compile bugs * fix bugs when merge * fix compiler bugs * fix bugs when run test_flatten_api * fix bugs when run test * Revert "use flat_hash_map and small_vector in kernel factory" This reverts commit 2309149. * Move cpu, cuda and other device code into kernels (PaddlePaddle#15) * fill_any_like kernel refactor * remove useless code of full_like c++ api * Support Scalar in Tensor Compute Library * add scalar in dygraph and static graph mode * keep the basic type for attr, instead of using scalar for all * merge the code * start refactor matmul * move cpu, cuda and other device modules into kernels * merge code * polish code in operator.cc * Perfect unitests (PaddlePaddle#16) * perfect unittest * update license * replace with flat_hash_map, small_vector (PaddlePaddle#19) * fix small_vector build error on windows platform * replace with flat_hash_map, small_vector * remove todo * Perfect unitests (PaddlePaddle#20) * perfect unittest * update license * fix bug when run tcmpt_utils_test * refactor execution adapting impl * fix insert conflit * Fix CI bug of test_yolov3 (PaddlePaddle#21) * fill_any_like kernel refactor * remove useless code of full_like c++ api * Support Scalar in Tensor Compute Library * add scalar in dygraph and static graph mode * keep the basic type for attr, instead of using scalar for all * merge the code * start refactor matmul * move cpu, cuda and other device modules into kernels * merge code * polish code in operator.cc * Fix CI bug of test_yolov3 * add the tensor base class, test=develop (PaddlePaddle#17) * update the tensor base class, test=develop * remove two funcs, test=develop * update the error msg, test=develop Co-authored-by: Chen Weihang <chenweihang@baidu.com> * [no-verify] commit backend and tensor signature changes * Rename tcmpt to pten (PaddlePaddle#23) * rename tcmpt to pten * update omitted files for rename to pten * update omitted file for rename to pten * remove k of all enum var * remove kernel_instantiate (PaddlePaddle#26) * remove symbols and spatial_tensor * change common to functions * readd share tensor impl methods * add a candidate dense tensor class, test=develop (PaddlePaddle#28) * change all Pt to Pten * resolve conflit with xiaowei * Op2functor opt1 (PaddlePaddle#27) * replace to small vector and change to const & * add std::move Co-authored-by: Chen Weihang <chenweihang@baidu.com> * polish kernel factory and kernel registry * fix operator test error msg mismatch * remove tensor signature and backend set member * move scalar and polish enforce * revert dtype layout change to fix error * fix enum operator override error * add several base unittests * add pten utils tests * polish some details * Dev/op2func refactor 3 (PaddlePaddle#30) * add a candidate dense tensor class, test=develop * remove TensorBase::backend(), test=develop * remove some ops, test=develop * cherry-pick the pr of tensor meta, test=develop * moves the dense tensor and some ops, test=develop * update the linalg operator, test=develop * update other operators, test=develop * fix errors, test=develop * fix bugs, test=develop * try to resolve the problem of windows ci, test=develop * updates codes, test=develop * fix the tensor_utils.cc, test=develop * modify the dense tensor, test=develop * fix the data type, test=develop Co-authored-by: shixiaowei02 <39303645+Shixiaowei02@users.noreply.github.com> * polish some details * polish kernel signature details * fix a bug about offsets of the tensor, test=develop (PaddlePaddle#31) Co-authored-by: shixiaowei02 <39303645+Shixiaowei02@users.noreply.github.com> * polish some details Co-authored-by: chentianyu03 <ctychentianyu@gmail.com> Co-authored-by: zyfncg <1370305206@qq.com> Co-authored-by: YuanRisheng <yuanrisheng@baidu.com> Co-authored-by: 石晓伟 <39303645+Shixiaowei02@users.noreply.github.com>

chenwhql added 7 commits July 9, 2021 11:50

initial tensor design & sign kernel demo

3f545c4

add move constructor for meta & add lodtensor

1f4ea40

add dirs & sign xpu kernel

44bf926

add mean cpu&cuda kernel impl

b20689d

move sign & mean xpu & npu kernel

79d2a1a

add selected_rows basic impl

434136f

refactor design, BaseTensor to DenseTensor, etc.

6c6ee22

chenwhql added 12 commits July 27, 2021 08:57

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

013c3fb

… op2func_refactor

add scale mkldnn kernel

33bba06

polish xpu & npu impl details

d895a11

fix mkldnn reuse compile failed

62ebf01

change tensor operation lib name

7c09726

resolve conflit with develop

7ae7f2f

rename util filename

288efc2

add more comments

be3ddd5

change TensorImplInterface to TensorInterface

3386c49

add kernel key and factory

4ef6be5

remove MKLDNNTensorMeta, add MKLDNNDenseTensor

b69066e

resolve conflict with develop

1d4f90e

chenwhql changed the title ~~[Demo for Discussion] The Tensor Design of Tensor Operation Library~~ [WIP] The Tensor Design of Tensor Operation Library Aug 5, 2021

chenwhql added 9 commits August 5, 2021 07:01

change XXDeviceContext to XXContext

c732d57

add base kernel registrar utils & test on sign

374345f

resolve conflict with develop

bbb6473

replace boost::any by paddle::any

0e18ff4

fix several ci failed

805896b

fix npu compile error

fc4442b

add ordered map util

cefe30a

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

aa3e79b

… op2func_refactor

fix multiple ordered_map compile errors

a1753a0

Shixiaowei02 reviewed Oct 27, 2021

View reviewed changes

polish some details

a83e9c7

Shixiaowei02 previously approved these changes Oct 28, 2021

View reviewed changes

zyfncg mentioned this pull request Oct 28, 2021

Add matmul_v2 kernel in pten #36844

Merged

jeff41404 previously approved these changes Oct 29, 2021

View reviewed changes

Superjomn previously approved these changes Oct 29, 2021

View reviewed changes

Avin0323 previously approved these changes Oct 30, 2021

View reviewed changes

Merge branch 'develop' into op2func_refactor

e9a64fa

chenwhql dismissed stale reviews from Avin0323, Superjomn, jeff41404, and Shixiaowei02 via e9a64fa November 1, 2021 02:16

Superjomn approved these changes Nov 1, 2021

View reviewed changes

phlrain self-requested a review November 1, 2021 07:12

phlrain approved these changes Nov 1, 2021

View reviewed changes

raindrops2sea approved these changes Nov 1, 2021

View reviewed changes

Avin0323 approved these changes Nov 1, 2021

View reviewed changes

XiaoguangHu01 approved these changes Nov 1, 2021

View reviewed changes

wanghuancoder approved these changes Nov 1, 2021

View reviewed changes

chenwhql merged commit b9fdd3b into PaddlePaddle:develop Nov 1, 2021

zyfncg mentioned this pull request Nov 2, 2021

[PTen] Add full kernel in pten (incomplete) #36930

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Paddle Tensor Operation Library initial implementation #34425

Paddle Tensor Operation Library initial implementation #34425

chenwhql commented Jul 27, 2021 •

edited

Loading

paddle-bot-old bot commented Jul 27, 2021

Shixiaowei02 Oct 27, 2021

Shixiaowei02 Oct 27, 2021

Shixiaowei02 Oct 27, 2021

Shixiaowei02 Oct 27, 2021

Shixiaowei02 Oct 27, 2021

Shixiaowei02 Oct 27, 2021

Shixiaowei02 Oct 27, 2021

Shixiaowei02 Oct 27, 2021

Shixiaowei02 Oct 27, 2021

jeff41404 left a comment

Superjomn left a comment

Avin0323 left a comment

Superjomn left a comment

Avin0323 left a comment

		@@ -0,0 +1,10 @@
		cc_test(pten_backend_test SRCS backend_test.cc DEPS gtest)

		@@ -0,0 +1,5 @@
		cc_library(math_cpu SRCS math.cc DEPS dense_tensor kernel_context kernel_factory eigen_function)

Paddle Tensor Operation Library initial implementation #34425

Paddle Tensor Operation Library initial implementation #34425

Conversation

chenwhql commented Jul 27, 2021 • edited Loading

PR types

PR changes

Describe

一、背景

二、目标

三、设计

四、现状说明

paddle-bot-old bot commented Jul 27, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jeff41404 left a comment

Choose a reason for hiding this comment

Superjomn left a comment

Choose a reason for hiding this comment

Avin0323 left a comment

Choose a reason for hiding this comment

Superjomn left a comment

Choose a reason for hiding this comment

Avin0323 left a comment

Choose a reason for hiding this comment

chenwhql commented Jul 27, 2021 •

edited

Loading