tensorsplit_op #7258

lcylcy · 2022-01-14T07:31:18Z

CLAassistant · 2022-01-14T07:31:22Z

All committers have signed the CLA.

wyushun · 2022-01-24T02:21:51Z

oneflow/core/functional/impl/math_functor.cpp

+  TensorSplitVecFunctor() = default;
+  Maybe<TensorTuple> operator()(const std::shared_ptr<one::Tensor>& input, 
+                           const std::vector<int32_t>& indices_or_sections,
+                           const int32_t& dim) const {


参考上个pr的comment：#7275 (comment)

wyushun · 2022-01-24T02:28:34Z

oneflow/core/functional/impl/math_functor.cpp

+    std::vector<int64_t> stop(ndim);
+    std::vector<int64_t> step(ndim, 1);
+    for(int32_t i=0; i<ndim; i++){
+       stop[i] = input->shape()->At(i);


one::Tensor有个更简单的接口：input->dim(i)

reference：https://github.com/Oneflow-Inc/oneflow/blob/master/oneflow/core/framework/tensor.h#L49

wyushun · 2022-01-24T02:29:02Z

oneflow/core/functional/impl/math_functor.cpp

+       output[i] = JUST(Slice(input, start, stop, step));
+       start[pos_dim] = end_idx;
+    }
+    stop[pos_dim] = input->shape()->At(ndim-1);


同上：https://github.com/Oneflow-Inc/oneflow/pull/7258/files#r790378085

wyushun · 2022-01-24T02:30:13Z

oneflow/core/functional/impl/math_functor.cpp

+                                const int32_t& indices_or_sections,
+                                const int32_t& dim) const {


同上：https://github.com/Oneflow-Inc/oneflow/pull/7258/files#r790376430

wyushun · 2022-01-24T02:30:59Z

oneflow/core/functional/impl/math_functor.cpp

+    std::vector<int64_t> stop(ndim);
+    std::vector<int64_t> step(ndim, 1);
+    for(int32_t i=0; i<ndim; i++){
+       stop[i] = input->shape()->At(i);


同上，https://github.com/Oneflow-Inc/oneflow/pull/7258/files#r790378085

wyushun · 2022-01-24T02:32:22Z

oneflow/core/functional/impl/math_functor.cpp

+  HsplitIntFunctor() = default;
+  Maybe<TensorTuple> operator()(const std::shared_ptr<one::Tensor>& input, 
+                                const int32_t& indices_or_sections) const {
+    int32_t ndim = input->shape()->NumAxes();


有更简单的接口：https://github.com/Oneflow-Inc/oneflow/blob/master/oneflow/core/framework/tensor.h#L51

wyushun · 2022-01-24T02:33:23Z

oneflow/core/functional/impl/math_functor.cpp

+ public:
+  HsplitIntFunctor() = default;
+  Maybe<TensorTuple> operator()(const std::shared_ptr<one::Tensor>& input, 
+                                const int32_t& indices_or_sections) const {


同上：https://github.com/Oneflow-Inc/oneflow/pull/7258/files#r790376430

wyushun · 2022-01-24T02:33:56Z

oneflow/core/functional/impl/math_functor.cpp

+  HsplitVecFunctor() = default;
+  Maybe<TensorTuple> operator()(const std::shared_ptr<one::Tensor>& input, 
+                                const std::vector<int32_t>& indices_or_sections) const {
+    int32_t ndim = input->shape()->NumAxes();


同上：https://github.com/Oneflow-Inc/oneflow/pull/7258/files#r790379044

wyushun · 2022-01-24T02:34:45Z

oneflow/core/functional/impl/math_functor.cpp

+ public:
+  VsplitIntFunctor() = default;
+  Maybe<TensorTuple> operator()(const std::shared_ptr<one::Tensor>& input, 
+                                const int32_t& indices_or_sections) const {


同上：https://github.com/Oneflow-Inc/oneflow/pull/7258/files#r790379044

wyushun · 2022-01-24T02:35:15Z

oneflow/core/functional/impl/math_functor.cpp

+    int32_t ndim = input->shape()->NumAxes();
+    CHECK_OR_RETURN(ndim>=2)<<"torch.vsplit requires a tensor with at least 2 dimension, but got a tensor with "<<ndim <<" dimensions!";
+    CHECK_OR_RETURN(indices_or_sections>0) << "indices_or_sections must greater than 0";
+    CHECK_OR_RETURN(input->shape()->At(0)% indices_or_sections == 0) << "torch.vsplit attempted to split along dimension " << 0 


consider input->dim()

wyushun · 2022-01-24T02:35:38Z

oneflow/core/functional/impl/math_functor.cpp

+  VsplitVecFunctor() = default;
+  Maybe<TensorTuple> operator()(const std::shared_ptr<one::Tensor>& input, 
+                                const std::vector<int32_t>& indices_or_sections) const {
+    int32_t ndim = input->shape()->NumAxes();


consider input->ndim()

wyushun · 2022-01-24T02:36:45Z

python/oneflow/test/modules/test_hsplit.py

+class TestHsplitVec(flow.unittest.TestCase):
+    @autotest(check_graph=False)
+    def test_flow_hsplit_vec(test_case):
+        device = random_device()
+        x = random_pytorch_tensor(
+            ndim=4,
+            dim1=random(3, 6),
+            dim2=random(3, 6),
+            dim3=random(3, 6),
+            dim4=random(3, 6),
+        ).to(device)
+        z = torch.hsplit(x, (1,2))
+        return z[0]
+
+class TestHsplitInt(flow.unittest.TestCase):
+    @autotest(check_graph=False)
+    def test_flow_hsplit_int(test_case):
+        device = random_device()
+        x = random_pytorch_tensor(
+            ndim=4,
+            dim1=random(3, 6),
+            dim2=random(3, 6),
+            dim3=random(3, 6),
+            dim4=random(3, 6),
+        ).to(device)
+        split = random(1, 3).to(int)
+        z = torch.hsplit(x, split)
+        return z[0]
+
+


参考一下这个：#7275 (comment)

wyushun · 2022-01-24T02:37:07Z

python/oneflow/test/modules/test_tensor_split.py

+class TestTorchSplitVec(flow.unittest.TestCase):
+    @autotest(check_graph=False)
+    def test_flow_tensor_split_vec(test_case):
+        device = random_device()
+        x = random_pytorch_tensor(
+            ndim=4,
+            dim1=random(3, 6),
+            dim2=random(3, 6),
+            dim3=random(3, 6),
+            dim4=random(3, 6),
+        ).to(device)
+        dim = random(-3, 3).to(int)
+        z = torch.tensor_split(x, (1,2),dim)
+        return z[0]
+
+class TestTorchSplitInt(flow.unittest.TestCase):
+    @autotest(check_graph=False)
+    def test_flow_tensor_split_int(test_case):
+        device = random_device()
+        x = random_pytorch_tensor(
+            ndim=4,
+            dim1=random(3, 6),
+            dim2=random(3, 6),
+            dim3=random(3, 6),
+            dim4=random(3, 6),
+        ).to(device)
+        split = random(-3, 3).to(int)
+        dim = random(-3, 3).to(int)
+        z = torch.tensor_split(x, split,dim)
+        return z[0]


wyushun · 2022-01-24T02:37:17Z

python/oneflow/test/modules/test_vsplit.py

+class TestVsplitVec(flow.unittest.TestCase):
+    @autotest(check_graph=False)
+    def test_flow_vsplit_vec(test_case):
+        device = random_device()
+        x = random_pytorch_tensor(
+            ndim=4,
+            dim1=random(3, 6),
+            dim2=random(3, 6),
+            dim3=random(3, 6),
+            dim4=random(3, 6),
+        ).to(device)
+        z = torch.vsplit(x, (1,2))
+        return z[0]
+
+class TestVsplitInt(flow.unittest.TestCase):
+    @autotest(check_graph=False)
+    def test_flow_vsplit_int(test_case):
+        device = random_device()
+        x = random_pytorch_tensor(
+            ndim=4,
+            dim1=random(3, 6),
+            dim2=random(3, 6),
+            dim3=random(3, 6),
+            dim4=random(3, 6),
+        ).to(device)
+        split = random(1, 3).to(int)
+        z = torch.vsplit(x, split)
+        return z[0]


wyushun

review done，写的太棒了，我写了一些comments，大多都和上个pr（implement as strided）的雷同，你看情况自己酌情修改哈，我直接给你approve了，这样可以尽可能提高效率，你自己做足测试保证正确性就好～ @lcylcy

lcylcy · 2022-01-24T02:46:16Z

好的

github-actions · 2022-01-27T02:25:18Z

Code got formatted by CI. Please request CI again if you still want to have this PR merged. If the PR is from a forked repo, please download the patch files from the GitHub Actions web page and apply them locally.

github-actions · 2022-01-27T05:08:35Z

CI failed when running job: cuda-module. PR label automerge has been removed

github-actions · 2022-01-27T07:48:07Z

Speed stats:

GPU Name: GeForce GTX 1080 

✔️ OneFlow resnet50 time: 136.6ms (= 13656.7ms / 100, input_shape=[16, 3, 224, 224])
PyTorch resnet50 time: 139.4ms (= 13943.8ms / 100, input_shape=[16, 3, 224, 224])
✔️ Relative speed: 1.02 (= 139.4ms / 136.6ms)

✔️ OneFlow resnet50 time: 78.6ms (= 7855.6ms / 100, input_shape=[8, 3, 224, 224])
PyTorch resnet50 time: 83.6ms (= 8357.3ms / 100, input_shape=[8, 3, 224, 224])
✔️ Relative speed: 1.06 (= 83.6ms / 78.6ms)

OneFlow resnet50 time: 52.9ms (= 10581.3ms / 200, input_shape=[4, 3, 224, 224])
PyTorch resnet50 time: 57.4ms (= 11486.2ms / 200, input_shape=[4, 3, 224, 224])
✔️ Relative speed: 1.09 (= 57.4ms / 52.9ms)

OneFlow resnet50 time: 41.7ms (= 8343.9ms / 200, input_shape=[2, 3, 224, 224])
PyTorch resnet50 time: 48.0ms (= 9597.2ms / 200, input_shape=[2, 3, 224, 224])
✔️ Relative speed: 1.15 (= 48.0ms / 41.7ms)

OneFlow resnet50 time: 40.9ms (= 8175.1ms / 200, input_shape=[1, 3, 224, 224])
PyTorch resnet50 time: 38.1ms (= 7614.3ms / 200, input_shape=[1, 3, 224, 224])
✔️ Relative speed: 0.93 (= 38.1ms / 40.9ms)

✔️ OneFlow resnet50 time: 148.9ms (= 14888.1ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 158.8ms (= 15884.0ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.07 (= 158.8ms / 148.9ms)

OneFlow resnet50 time: 90.0ms (= 9004.3ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 101.2ms (= 10117.9ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.12 (= 101.2ms / 90.0ms)

OneFlow resnet50 time: 65.7ms (= 13137.3ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 72.8ms (= 14555.0ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.11 (= 72.8ms / 65.7ms)

OneFlow resnet50 time: 52.6ms (= 10526.3ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 62.3ms (= 12454.1ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.18 (= 62.3ms / 52.6ms)

OneFlow resnet50 time: 57.4ms (= 11474.0ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 57.9ms (= 11583.3ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.01 (= 57.9ms / 57.4ms)

first

b9c5bb8

lcylcy requested review from doombeaker and BBuf January 14, 2022 07:31

lcylcy requested review from daquexian, hjchen2 and jackalcooper as code owners January 14, 2022 07:31

hsplit,vsplit

41595ed

wyushun reviewed Jan 24, 2022

View reviewed changes

wyushun approved these changes Jan 24, 2022

View reviewed changes

lcylcy changed the title ~~first~~ tensorsplit_op Jan 24, 2022

luqiang-guo approved these changes Jan 26, 2022

View reviewed changes

lcylcy added 3 commits January 27, 2022 10:10

revise

b9f34a9

confict

ef0770b

confict

970edcd

lcylcy requested a review from oneflow-ci-bot January 27, 2022 02:22

lcylcy removed the request for review from oneflow-ci-bot January 27, 2022 02:22

lcylcy added op automerge enhancement labels Jan 27, 2022

lcylcy requested a review from oneflow-ci-bot January 27, 2022 02:23

lcylcy requested review from oneflow-ci-bot and removed request for oneflow-ci-bot January 27, 2022 02:26

oneflow-ci-bot requested review from oneflow-ci-bot and removed request for oneflow-ci-bot January 27, 2022 03:52

github-actions bot removed the automerge label Jan 27, 2022

oneflow-ci-bot removed their request for review January 27, 2022 05:12

docs

b41b8b2

lcylcy force-pushed the lcy_tensorsplit branch from 7c016a3 to b41b8b2 Compare January 27, 2022 06:18

lcylcy and others added 2 commits January 27, 2022 14:20

formatted

7bce539

Merge branch 'master' into lcy_tensorsplit

a946188

lcylcy added the automerge label Jan 27, 2022

lcylcy requested a review from oneflow-ci-bot January 27, 2022 06:21

oneflow-ci-bot removed their request for review January 27, 2022 07:48

oneflow-ci-bot merged commit e11cd60 into master Jan 27, 2022

oneflow-ci-bot deleted the lcy_tensorsplit branch January 27, 2022 07:50

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tensorsplit_op #7258

tensorsplit_op #7258

lcylcy commented Jan 14, 2022 •

edited

Loading

CLAassistant commented Jan 14, 2022 •

edited

Loading

wyushun Jan 24, 2022

wyushun Jan 24, 2022

wyushun Jan 24, 2022 •

edited

Loading

wyushun Jan 24, 2022

wyushun Jan 24, 2022

wyushun Jan 24, 2022 •

edited

Loading

wyushun Jan 24, 2022

wyushun Jan 24, 2022

wyushun Jan 24, 2022

wyushun Jan 24, 2022

wyushun Jan 24, 2022

wyushun Jan 24, 2022

wyushun Jan 24, 2022

wyushun Jan 24, 2022

wyushun left a comment •

edited

Loading

lcylcy commented Jan 24, 2022

github-actions bot commented Jan 27, 2022

github-actions bot commented Jan 27, 2022

github-actions bot commented Jan 27, 2022

		const int32_t& indices_or_sections,
		const int32_t& dim) const {

tensorsplit_op #7258

tensorsplit_op #7258

Conversation

lcylcy commented Jan 14, 2022 • edited Loading

CLAassistant commented Jan 14, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wyushun Jan 24, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wyushun Jan 24, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wyushun left a comment • edited Loading

Choose a reason for hiding this comment

lcylcy commented Jan 24, 2022

github-actions bot commented Jan 27, 2022

github-actions bot commented Jan 27, 2022

github-actions bot commented Jan 27, 2022

lcylcy commented Jan 14, 2022 •

edited

Loading

CLAassistant commented Jan 14, 2022 •

edited

Loading

wyushun Jan 24, 2022 •

edited

Loading

wyushun Jan 24, 2022 •

edited

Loading

wyushun left a comment •

edited

Loading