add glu op #6065

Ikkyu321 · 2021-08-26T06:28:27Z

对齐torch.nn.functional.glu算子
门控线性单元

hengzi · 2021-08-26T06:37:16Z

python/oneflow/nn/modules/activation.py

+    .. code-block:: python
+
+        >>> import oneflow as flow
+        >>> import oneflow.nn.functional as nn


一般不这么写吧

一般不这么写吧

已修改

hengzi · 2021-08-26T06:38:03Z

docs/source/oneflow.rst

@@ -50,7 +50,8 @@ oneflow
            fmod, 
            gather, 
            gather_nd, 
-            gelu, 
+            gelu,
+            glu, 


不需要导出oneflow.glu

不需要导出oneflow.glu

已删除

hengzi · 2021-08-26T06:40:28Z

docs/source/tensor.rst

@@ -56,7 +56,8 @@ OneFlow Tensor Class
            gather, 
            ge, 
            gelu, 
-            get_device, 
+            get_device,
+            glu, 


不需要导出oneflow.tensor.glu

不需要导出oneflow.tensor.glu

已删除

hengzi · 2021-08-26T06:40:58Z

python/oneflow/nn/modules/activation.py

+    return y[0] * flow.sigmoid(y[1])
+
+
+@register_tensor_op("glu")


不需要

已删除

…dev_glu_op

…nto dev_glu_op

…dev_glu_op

…nto dev_glu_op

doombeaker · 2021-08-27T07:45:24Z

python/oneflow/nn/modules/activation.py

@@ -1162,6 +1162,56 @@ def softsign_op_tensor(x):
    return Softsign()(x)


+def glu_op(x, dim=-1):


Suggested change

def glu_op(x, dim=-1):

def glu_op(input, dim=-1):

形参名字没对齐

doombeaker · 2021-08-27T07:45:25Z

python/oneflow/nn/modules/activation.py

@@ -1162,6 +1162,56 @@ def softsign_op_tensor(x):
    return Softsign()(x)


+def glu_op(x, dim=-1):


Suggested change

def glu_op(x, dim=-1):

def glu_op(input, dim=-1):

形参名字没对齐

Flowingsun007 · 2021-08-27T07:48:30Z

python/oneflow/nn/modules/activation.py

+    y = x.split([nc, nc], dim)
+
+    return y[0] * flow.sigmoid(y[1])
+


需要导出GLU的module：class GLU(Module) （pytorch源码里也导出了，可参考）

函数逻辑可以注册op/kernel，并通过functor接口导出，flow.F.glu(xxx)

需要导出GLU的module：class GLU(Module) （pytorch源码里也导出了，可参考）

GLU已导出，在c++层使用算子拼接实现

python/oneflow/nn/modules/activation.py

doombeaker · 2021-08-27T07:54:54Z

python/oneflow/nn/modules/activation.py

+                [4.9954, 5.9980]], dtype=oneflow.float32)
+
+    """
+    ndim = len(x.shape)


这些逻辑都放到 C++ 里面去吧。让 python 端处理的工作尽可能少。

C++端利用 functor，也可以很方便地拼接算子了。

稍微要注意的是，y[0] 在 C++ 里没有重载，需要使用 slice functor 取出内容，再用 mul 的 functor 和 sigmoid 做运算。

可以参考 https://github.com/Oneflow-Inc/oneflow/blob/master/oneflow/core/functional/impl/array_functor.cpp#L352

python/oneflow/nn/modules/activation.py

hengzi · 2021-09-07T07:17:19Z

python/oneflow/test/modules/test_glu.py

+        m = torch.nn.functional.glu
+        x = random_pytorch_tensor(ndim=3, dim0=2, dim1=4, dim2=6).to(device)
+        y = m(x, dim)
+        return y


导出 nn.GLU接口，应该也要加上对应的测试

…dev_glu_op

…nto dev_glu_op

github-actions · 2021-09-07T15:57:35Z

Speed stats:

GPU Name: GeForce GTX 1080 

OneFlow resnet50 time: 128.5ms (= 6424.9ms / 50, input_shape=[16, 3, 224, 224])
PyTorch resnet50 time: 140.2ms (= 7011.9ms / 50, input_shape=[16, 3, 224, 224])
Relative speed: 1.09 (= 140.2ms / 128.5ms)

OneFlow resnet50 time: 74.6ms (= 3730.8ms / 50, input_shape=[8, 3, 224, 224])
PyTorch resnet50 time: 83.7ms (= 4184.2ms / 50, input_shape=[8, 3, 224, 224])
Relative speed: 1.12 (= 83.7ms / 74.6ms)

OneFlow resnet50 time: 47.7ms (= 2386.7ms / 50, input_shape=[4, 3, 224, 224])
PyTorch resnet50 time: 58.5ms (= 2924.4ms / 50, input_shape=[4, 3, 224, 224])
Relative speed: 1.23 (= 58.5ms / 47.7ms)

OneFlow resnet50 time: 40.9ms (= 2044.0ms / 50, input_shape=[2, 3, 224, 224])
PyTorch resnet50 time: 48.8ms (= 2442.0ms / 50, input_shape=[2, 3, 224, 224])
Relative speed: 1.19 (= 48.8ms / 40.9ms)

OneFlow resnet50 time: 40.6ms (= 2030.7ms / 50, input_shape=[1, 3, 224, 224])
PyTorch resnet50 time: 44.4ms (= 2221.5ms / 50, input_shape=[1, 3, 224, 224])
Relative speed: 1.09 (= 44.4ms / 40.6ms)

OneFlow resnet50 time: 153.8ms (= 7690.1ms / 50, input_shape=[16, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 160.0ms (= 8001.8ms / 50, input_shape=[16, 3, 224, 224], ddp, world size=2)
Relative speed: 1.04 (= 160.0ms / 153.8ms)

OneFlow resnet50 time: 103.7ms (= 5185.7ms / 50, input_shape=[8, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 101.8ms (= 5087.8ms / 50, input_shape=[8, 3, 224, 224], ddp, world size=2)
Relative speed: 0.98 (= 101.8ms / 103.7ms)

OneFlow resnet50 time: 74.8ms (= 3739.5ms / 50, input_shape=[4, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 74.0ms (= 3698.2ms / 50, input_shape=[4, 3, 224, 224], ddp, world size=2)
Relative speed: 0.99 (= 74.0ms / 74.8ms)

OneFlow resnet50 time: 72.5ms (= 3625.5ms / 50, input_shape=[2, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 74.1ms (= 3703.5ms / 50, input_shape=[2, 3, 224, 224], ddp, world size=2)
Relative speed: 1.02 (= 74.1ms / 72.5ms)

OneFlow resnet50 time: 68.6ms (= 3431.1ms / 50, input_shape=[1, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 61.3ms (= 3065.5ms / 50, input_shape=[1, 3, 224, 224], ddp, world size=2)
Relative speed: 0.89 (= 61.3ms / 68.6ms)

* Primitive (#6183) * Add Primitive * #ifdef WITH_CUDA Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> * Disable implicit boxing when parallel num eq one (#6188) * mv_boxing_folder_to_core * minor fix * disable_implicit_boxing_when_parallel_num_eq_one * Update eager_consistent_op_interpreter.cpp Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> * Lazy support Scalar (#6181) Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> * Fix LayerNorm check bug (#6196) * fix(Layernorm): fix check bug * fix judge whether cpu or not Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> * add glu op (#6065) * add glu op * del glu_op export，align with torch * mod glu_op * mov op logic to C++ * Solve problems * solve conflict * delete gradient functor * add ndim check * add GLU test * delete blank line * format Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> Co-authored-by: Zhenhua <huangzhenhua@zhejianglab.com> * Primitive based copy task node (#6195) Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> * KernelState (#6198) Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> * container_util: fix VectorAt, remove useless MutMapAt (#6172) * fcontainer_util: fix VectorAt, remove useless MutMapAt * fcontainer_util: format * MapAt: add default value version * format Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> * Refine StreamContext (#6191) Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> * Cpu symetric s to s (#6153) * mv_boxing_folder_to_core * minor fix * cpu_symetric_s_to_s * add test case * auto format by CI * minor fix * refine * Update eager_nccl_kernels.cpp * minor fix * fix bug * minor fix * Update oneflow/user/kernels/eager_nccl_kernels.cpp Co-authored-by: daquexian <daquexian566@gmail.com> * Update eager_nccl_kernels.cpp * Update eager_nccl_kernels.cpp * minor fix * Update eager_nccl_kernels.cpp Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org> Co-authored-by: daquexian <daquexian566@gmail.com> * fix bug (#6197) Co-authored-by: Yinggang Wang <wyg19970408@gmail.com> Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> * fix consistent tensor zeros (#6202) Signed-off-by: daquexian <daquexian566@gmail.com> Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> * [Feat.] nn.Graph support grad acc with input/output tensor (#6155) * nn.Graph support grad acc with input/output tensor * dirty pass grad acc * revert tensor.backward hack * fix indent * default S0 -> B * Pack op/kernel support scalar input * nn.Graph output pack support loss scalar * add test script * pass test * Lazy build output eager tensors after job complete * non scalar output test Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> * Dev eliminate gcc warnings (#6199) * fix gcc warning * refine * fix comment Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> * StreamContextAdapter (#6205) * StreamContextAdapter * refine Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> * Autotest generate input tensor (#6206) * Add tensor yaml, support export tensor functional api. * refine * Remove packed functor signature * remove unused file * Refine * refine * add activation op import * reinit oneflow init.py * add oneflow abs and exp * add oneflow abs and exp * add acos * add arccosh * add more op * add more ops * add more op * add more ops * add log1p * add more smaples * add more ops * add more ops * add more ops * add more ops * Complete tensor functional apis. * Fix pybind call * add more ops * add ops done * Add target of_functional_tensor_obj * Disable throw visibility warnings * fix target link * fix * fix incorrect use of flow.Tensor. * Fix error merge * fix * fix add unittest * refine * refine * fix * fix * add tensor doc * auto format by CI * refine * Fix * Add doc for python function * refine * add tensor method docstring * fix some bug * fix docs bug * Fix * auto format by CI * Tensor->tensor * Tensor->tensor * refine Tensor->tensor * fix * fix * fix * fix conflict * fix bug * fix ci bug * fix * delete diag op * fix conflict * Fix segment * fix * merge * autotest framework generate input tensor * autotest framework generate input tensor * fix bug * fix impl bug * refine * refine * refine * fix * fix * fix comments * delete useless * fix ci error * fix ci error Co-authored-by: hjchen2 <chenhoujiangcug@gmail.com> Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org> Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> * Cleanup KernelUtil (#6212) Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> * Rename flow to oneflow in user hint (#6190) * style(*): rename flow to oneflow in user hint * fix(*): fix doctest * auto format by CI * remove ddp speed test Signed-off-by: daquexian <daquexian566@gmail.com> Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org> Co-authored-by: daquexian <daquexian566@gmail.com> * merg and refactor * refact code * add io identity for activation checkpointing Co-authored-by: Juncheng <liujuncheng1022@gmail.com> Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> Co-authored-by: binbinHan <han_binbin@163.com> Co-authored-by: cheng cheng <472491134@qq.com> Co-authored-by: Yinggang Wang <wyg19970408@gmail.com> Co-authored-by: QiangX-man <87475073+QiangX-man@users.noreply.github.com> Co-authored-by: Zhenhua <huangzhenhua@zhejianglab.com> Co-authored-by: Twice <i@twice.moe> Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org> Co-authored-by: daquexian <daquexian566@gmail.com> Co-authored-by: ZZK <42901638+MARD1NO@users.noreply.github.com> Co-authored-by: Luyang <flowingsun007@163.com> Co-authored-by: Xiaoyu Zhang <35585791+BBuf@users.noreply.github.com> Co-authored-by: hjchen2 <chenhoujiangcug@gmail.com>

* Add test_graph_activation_checkpoint.py * Fea/checkpoint stop identity (#6216) * Primitive (#6183) * Add Primitive * #ifdef WITH_CUDA Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> * Disable implicit boxing when parallel num eq one (#6188) * mv_boxing_folder_to_core * minor fix * disable_implicit_boxing_when_parallel_num_eq_one * Update eager_consistent_op_interpreter.cpp Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> * Lazy support Scalar (#6181) Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> * Fix LayerNorm check bug (#6196) * fix(Layernorm): fix check bug * fix judge whether cpu or not Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> * add glu op (#6065) * add glu op * del glu_op export，align with torch * mod glu_op * mov op logic to C++ * Solve problems * solve conflict * delete gradient functor * add ndim check * add GLU test * delete blank line * format Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> Co-authored-by: Zhenhua <huangzhenhua@zhejianglab.com> * Primitive based copy task node (#6195) Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> * KernelState (#6198) Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> * container_util: fix VectorAt, remove useless MutMapAt (#6172) * fcontainer_util: fix VectorAt, remove useless MutMapAt * fcontainer_util: format * MapAt: add default value version * format Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> * Refine StreamContext (#6191) Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> * Cpu symetric s to s (#6153) * mv_boxing_folder_to_core * minor fix * cpu_symetric_s_to_s * add test case * auto format by CI * minor fix * refine * Update eager_nccl_kernels.cpp * minor fix * fix bug * minor fix * Update oneflow/user/kernels/eager_nccl_kernels.cpp Co-authored-by: daquexian <daquexian566@gmail.com> * Update eager_nccl_kernels.cpp * Update eager_nccl_kernels.cpp * minor fix * Update eager_nccl_kernels.cpp Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org> Co-authored-by: daquexian <daquexian566@gmail.com> * fix bug (#6197) Co-authored-by: Yinggang Wang <wyg19970408@gmail.com> Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> * fix consistent tensor zeros (#6202) Signed-off-by: daquexian <daquexian566@gmail.com> Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> * [Feat.] nn.Graph support grad acc with input/output tensor (#6155) * nn.Graph support grad acc with input/output tensor * dirty pass grad acc * revert tensor.backward hack * fix indent * default S0 -> B * Pack op/kernel support scalar input * nn.Graph output pack support loss scalar * add test script * pass test * Lazy build output eager tensors after job complete * non scalar output test Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> * Dev eliminate gcc warnings (#6199) * fix gcc warning * refine * fix comment Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> * StreamContextAdapter (#6205) * StreamContextAdapter * refine Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> * Autotest generate input tensor (#6206) * Add tensor yaml, support export tensor functional api. * refine * Remove packed functor signature * remove unused file * Refine * refine * add activation op import * reinit oneflow init.py * add oneflow abs and exp * add oneflow abs and exp * add acos * add arccosh * add more op * add more ops * add more op * add more ops * add log1p * add more smaples * add more ops * add more ops * add more ops * add more ops * Complete tensor functional apis. * Fix pybind call * add more ops * add ops done * Add target of_functional_tensor_obj * Disable throw visibility warnings * fix target link * fix * fix incorrect use of flow.Tensor. * Fix error merge * fix * fix add unittest * refine * refine * fix * fix * add tensor doc * auto format by CI * refine * Fix * Add doc for python function * refine * add tensor method docstring * fix some bug * fix docs bug * Fix * auto format by CI * Tensor->tensor * Tensor->tensor * refine Tensor->tensor * fix * fix * fix * fix conflict * fix bug * fix ci bug * fix * delete diag op * fix conflict * Fix segment * fix * merge * autotest framework generate input tensor * autotest framework generate input tensor * fix bug * fix impl bug * refine * refine * refine * fix * fix * fix comments * delete useless * fix ci error * fix ci error Co-authored-by: hjchen2 <chenhoujiangcug@gmail.com> Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org> Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> * Cleanup KernelUtil (#6212) Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> * Rename flow to oneflow in user hint (#6190) * style(*): rename flow to oneflow in user hint * fix(*): fix doctest * auto format by CI * remove ddp speed test Signed-off-by: daquexian <daquexian566@gmail.com> Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org> Co-authored-by: daquexian <daquexian566@gmail.com> * merg and refactor * refact code * add io identity for activation checkpointing Co-authored-by: Juncheng <liujuncheng1022@gmail.com> Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> Co-authored-by: binbinHan <han_binbin@163.com> Co-authored-by: cheng cheng <472491134@qq.com> Co-authored-by: Yinggang Wang <wyg19970408@gmail.com> Co-authored-by: QiangX-man <87475073+QiangX-man@users.noreply.github.com> Co-authored-by: Zhenhua <huangzhenhua@zhejianglab.com> Co-authored-by: Twice <i@twice.moe> Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org> Co-authored-by: daquexian <daquexian566@gmail.com> Co-authored-by: ZZK <42901638+MARD1NO@users.noreply.github.com> Co-authored-by: Luyang <flowingsun007@163.com> Co-authored-by: Xiaoyu Zhang <35585791+BBuf@users.noreply.github.com> Co-authored-by: hjchen2 <chenhoujiangcug@gmail.com> * rm useless code * refien test * add skip * import os * fix test * auto format by CI Co-authored-by: strint <xiaoyulink@gmail.com> Co-authored-by: Juncheng <liujuncheng1022@gmail.com> Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> Co-authored-by: binbinHan <han_binbin@163.com> Co-authored-by: cheng cheng <472491134@qq.com> Co-authored-by: Yinggang Wang <wyg19970408@gmail.com> Co-authored-by: QiangX-man <87475073+QiangX-man@users.noreply.github.com> Co-authored-by: Zhenhua <huangzhenhua@zhejianglab.com> Co-authored-by: Twice <i@twice.moe> Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org> Co-authored-by: daquexian <daquexian566@gmail.com> Co-authored-by: ZZK <42901638+MARD1NO@users.noreply.github.com> Co-authored-by: Luyang <flowingsun007@163.com> Co-authored-by: Xiaoyu Zhang <35585791+BBuf@users.noreply.github.com> Co-authored-by: hjchen2 <chenhoujiangcug@gmail.com>

* add demo * add test draft * debug pipeline graph * add pp * add * support 0-d tensor spilt * add repr of conf for debug * add acc grad * add tag * hack stage id of to_consistent * Pipeline buffer pass stage id check log * FixPipelineStageIdPass interface * nn.Graph support grad acc with input/output tensor * dirty pass grad acc * Lazy support Scalar * add log * revert tensor.backward hack * fix indent * default S0 -> B * merge * Pack op/kernel support scalar input * nn.Graph output pack support loss scalar * add test script * add pack * pass test * Add test_graph_activation_checkpoint.py * Lazy build output eager tensors after job complete * non scalar output test * stale input * add debug * add * add test * add test * deal with empty kernel of_record tensor buffer * add test * refine test * New scope by stage id * rm debug log * finish test * half impl of stage id infer * Fix pipeline stage id by close max group * remove note * remove debug code * Fea/checkpoint stop identity (#6216) * Primitive (#6183) * Add Primitive * #ifdef WITH_CUDA Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> * Disable implicit boxing when parallel num eq one (#6188) * mv_boxing_folder_to_core * minor fix * disable_implicit_boxing_when_parallel_num_eq_one * Update eager_consistent_op_interpreter.cpp Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> * Lazy support Scalar (#6181) Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> * Fix LayerNorm check bug (#6196) * fix(Layernorm): fix check bug * fix judge whether cpu or not Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> * add glu op (#6065) * add glu op * del glu_op export，align with torch * mod glu_op * mov op logic to C++ * Solve problems * solve conflict * delete gradient functor * add ndim check * add GLU test * delete blank line * format Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> Co-authored-by: Zhenhua <huangzhenhua@zhejianglab.com> * Primitive based copy task node (#6195) Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> * KernelState (#6198) Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> * container_util: fix VectorAt, remove useless MutMapAt (#6172) * fcontainer_util: fix VectorAt, remove useless MutMapAt * fcontainer_util: format * MapAt: add default value version * format Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> * Refine StreamContext (#6191) Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> * Cpu symetric s to s (#6153) * mv_boxing_folder_to_core * minor fix * cpu_symetric_s_to_s * add test case * auto format by CI * minor fix * refine * Update eager_nccl_kernels.cpp * minor fix * fix bug * minor fix * Update oneflow/user/kernels/eager_nccl_kernels.cpp Co-authored-by: daquexian <daquexian566@gmail.com> * Update eager_nccl_kernels.cpp * Update eager_nccl_kernels.cpp * minor fix * Update eager_nccl_kernels.cpp Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org> Co-authored-by: daquexian <daquexian566@gmail.com> * fix bug (#6197) Co-authored-by: Yinggang Wang <wyg19970408@gmail.com> Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> * fix consistent tensor zeros (#6202) Signed-off-by: daquexian <daquexian566@gmail.com> Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> * [Feat.] nn.Graph support grad acc with input/output tensor (#6155) * nn.Graph support grad acc with input/output tensor * dirty pass grad acc * revert tensor.backward hack * fix indent * default S0 -> B * Pack op/kernel support scalar input * nn.Graph output pack support loss scalar * add test script * pass test * Lazy build output eager tensors after job complete * non scalar output test Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> * Dev eliminate gcc warnings (#6199) * fix gcc warning * refine * fix comment Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> * StreamContextAdapter (#6205) * StreamContextAdapter * refine Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> * Autotest generate input tensor (#6206) * Add tensor yaml, support export tensor functional api. * refine * Remove packed functor signature * remove unused file * Refine * refine * add activation op import * reinit oneflow init.py * add oneflow abs and exp * add oneflow abs and exp * add acos * add arccosh * add more op * add more ops * add more op * add more ops * add log1p * add more smaples * add more ops * add more ops * add more ops * add more ops * Complete tensor functional apis. * Fix pybind call * add more ops * add ops done * Add target of_functional_tensor_obj * Disable throw visibility warnings * fix target link * fix * fix incorrect use of flow.Tensor. * Fix error merge * fix * fix add unittest * refine * refine * fix * fix * add tensor doc * auto format by CI * refine * Fix * Add doc for python function * refine * add tensor method docstring * fix some bug * fix docs bug * Fix * auto format by CI * Tensor->tensor * Tensor->tensor * refine Tensor->tensor * fix * fix * fix * fix conflict * fix bug * fix ci bug * fix * delete diag op * fix conflict * Fix segment * fix * merge * autotest framework generate input tensor * autotest framework generate input tensor * fix bug * fix impl bug * refine * refine * refine * fix * fix * fix comments * delete useless * fix ci error * fix ci error Co-authored-by: hjchen2 <chenhoujiangcug@gmail.com> Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org> Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> * Cleanup KernelUtil (#6212) Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> * Rename flow to oneflow in user hint (#6190) * style(*): rename flow to oneflow in user hint * fix(*): fix doctest * auto format by CI * remove ddp speed test Signed-off-by: daquexian <daquexian566@gmail.com> Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org> Co-authored-by: daquexian <daquexian566@gmail.com> * merg and refactor * refact code * add io identity for activation checkpointing Co-authored-by: Juncheng <liujuncheng1022@gmail.com> Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> Co-authored-by: binbinHan <han_binbin@163.com> Co-authored-by: cheng cheng <472491134@qq.com> Co-authored-by: Yinggang Wang <wyg19970408@gmail.com> Co-authored-by: QiangX-man <87475073+QiangX-man@users.noreply.github.com> Co-authored-by: Zhenhua <huangzhenhua@zhejianglab.com> Co-authored-by: Twice <i@twice.moe> Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org> Co-authored-by: daquexian <daquexian566@gmail.com> Co-authored-by: ZZK <42901638+MARD1NO@users.noreply.github.com> Co-authored-by: Luyang <flowingsun007@163.com> Co-authored-by: Xiaoyu Zhang <35585791+BBuf@users.noreply.github.com> Co-authored-by: hjchen2 <chenhoujiangcug@gmail.com> * rm useless code * refien test * add skip * Fix bug of Multi-Client src tick output order * refactor * test pass * waning to info * import os * fix test * auto format by CI * refactor fix pipeline stage id pass algo * add refine test * insert identity when do pipeline * refine test * fix copy buffer * refien pipeline demo * format * Add input/output ctrl edge to DstSubTick for order io and callback_notify * add test scripts * remove note * add comment for using pipeling * add comment * rm useless to * avoid cpu run cuda test Co-authored-by: chengtbf <472491134@qq.com> Co-authored-by: peitingkuan <peitingkuan@163.com> Co-authored-by: oneflow-ci-bot <69100618+oneflow-ci-bot@users.noreply.github.com> Co-authored-by: Juncheng <liujuncheng1022@gmail.com> Co-authored-by: binbinHan <han_binbin@163.com> Co-authored-by: Yinggang Wang <wyg19970408@gmail.com> Co-authored-by: QiangX-man <87475073+QiangX-man@users.noreply.github.com> Co-authored-by: Zhenhua <huangzhenhua@zhejianglab.com> Co-authored-by: Twice <i@twice.moe> Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org> Co-authored-by: daquexian <daquexian566@gmail.com> Co-authored-by: ZZK <42901638+MARD1NO@users.noreply.github.com> Co-authored-by: Luyang <flowingsun007@163.com> Co-authored-by: Xiaoyu Zhang <35585791+BBuf@users.noreply.github.com> Co-authored-by: hjchen2 <chenhoujiangcug@gmail.com>

add glu op

cc24cdd

Ikkyu321 added api eager enhancement op labels Aug 26, 2021

hengzi reviewed Aug 26, 2021

View reviewed changes

Merge branch 'master' into dev_glu_op

74de30a

Ikkyu321 requested a review from hengzi August 26, 2021 08:46

Ikkyu321 and others added 13 commits August 26, 2021 17:19

Merge branch 'master' of https://github.com/Oneflow-Inc/oneflow into …

9e8e58b

…dev_glu_op

Merge branch 'master' into dev_glu_op

4ea4c15

del glu_op export，align with torch

a61d18f

Merge branch 'dev_glu_op' of https://github.com/Oneflow-Inc/oneflow i…

d769766

…nto dev_glu_op

Merge branch 'master' into dev_glu_op

e93cc42

Merge branch 'master' into dev_glu_op

5b62347

Merge branch 'master' into dev_glu_op

d78bee5

Merge branch 'master' of https://github.com/Oneflow-Inc/oneflow into …

520f61b

…dev_glu_op

Merge branch 'master' into dev_glu_op

0a21498

mod glu_op

d77709e

Merge branch 'master' of https://github.com/Oneflow-Inc/oneflow into …

32c613f

…dev_glu_op

Merge branch 'dev_glu_op' of https://github.com/Oneflow-Inc/oneflow i…

532081c

…nto dev_glu_op

Merge branch 'master' into dev_glu_op

61c041a

doombeaker reviewed Aug 27, 2021

View reviewed changes

Flowingsun007 reviewed Aug 27, 2021

View reviewed changes

doombeaker reviewed Aug 27, 2021

View reviewed changes

python/oneflow/nn/modules/activation.py Outdated Show resolved Hide resolved

doombeaker reviewed Aug 27, 2021

View reviewed changes

Ikkyu321 requested a review from doombeaker August 30, 2021 11:17

Ikkyu321 and others added 2 commits September 1, 2021 09:43

Merge branch 'master' into dev_glu_op

1b6f2cc

mov op logic to C++

b30f661

oneflow-ci-bot self-requested a review September 7, 2021 06:25

hengzi reviewed Sep 7, 2021

View reviewed changes

python/oneflow/nn/modules/activation.py Show resolved Hide resolved

hengzi reviewed Sep 7, 2021

View reviewed changes

Merge branch 'master' of https://github.com/Oneflow-Inc/oneflow into …

e42eea9

…dev_glu_op

hengzi removed the automerge label Sep 7, 2021

Ikkyu321 added 3 commits September 7, 2021 15:45

Merge branch 'dev_glu_op' of https://github.com/Oneflow-Inc/oneflow i…

197d7c7

…nto dev_glu_op

add GLU test

35ed600

delete blank line

2735a51

hengzi approved these changes Sep 7, 2021

View reviewed changes

hengzi added the automerge label Sep 7, 2021

hengzi and others added 2 commits September 7, 2021 15:59

format

96e185a

Merge branch 'master' into dev_glu_op

27c0b36

oneflow-ci-bot requested review from oneflow-ci-bot and removed request for oneflow-ci-bot September 7, 2021 08:37

Merge branch 'master' into dev_glu_op

9b9d673

oneflow-ci-bot requested review from oneflow-ci-bot and removed request for oneflow-ci-bot September 7, 2021 09:42

Merge branch 'master' into dev_glu_op

2058bfa

oneflow-ci-bot self-requested a review September 7, 2021 13:56

Merge branch 'master' into dev_glu_op

9556072

oneflow-ci-bot requested review from oneflow-ci-bot and removed request for oneflow-ci-bot September 7, 2021 14:46

oneflow-ci-bot merged commit da4d88b into master Sep 7, 2021

oneflow-ci-bot deleted the dev_glu_op branch September 7, 2021 15:59

BBuf mentioned this pull request Sep 28, 2021

OneFlow 算子对齐 PyTorch 完备计划推进表 #4936

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add glu op #6065

add glu op #6065

Ikkyu321 commented Aug 26, 2021 •

edited

Loading

hengzi Aug 26, 2021

Ikkyu321 Aug 27, 2021

hengzi Aug 26, 2021

Ikkyu321 Aug 27, 2021

hengzi Aug 26, 2021

Ikkyu321 Aug 27, 2021

hengzi Aug 26, 2021

Ikkyu321 Aug 27, 2021

doombeaker Aug 27, 2021

doombeaker Aug 27, 2021

Flowingsun007 Aug 27, 2021 •

edited

Loading

Flowingsun007 Aug 27, 2021

Ikkyu321 Sep 7, 2021

doombeaker Aug 27, 2021

hengzi Sep 7, 2021

github-actions bot commented Sep 7, 2021

		@@ -1162,6 +1162,56 @@ def softsign_op_tensor(x):
		return Softsign()(x)


		def glu_op(x, dim=-1):

add glu op #6065

add glu op #6065

Conversation

Ikkyu321 commented Aug 26, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Flowingsun007 Aug 27, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Sep 7, 2021

Ikkyu321 commented Aug 26, 2021 •

edited

Loading

Flowingsun007 Aug 27, 2021 •

edited

Loading