Update autotest framework #5520

daquexian · 2021-07-16T07:47:24Z

重构自动测试框架让 generator 可以组合。原来的以 1/3 的概率对有默认值参数不传参的行为现在需要显式触发，可以通过将 generator 设置为 random_or_nothing 或者 oneof(x, nothing(), possibility=2/3)、x | nothing() 等方式触发，generator 为 random、constant 等时不会再考虑 api 参数的默认值。

如果要跳过某一个参数的生成（例如新版 pytorch bn 的 device 和 dtype），直接将 generator 设置为 nothing() 即可

添加 torch_flow_dual_object.py，实现 from automated_test_util import * 之后，可以通过

    @autotest(auto_backward=False)
    def test_against_pytorch(test_case):
        m = torch.nn.Flatten(start_dim=random(1, 6) | nothing(),
                end_dim=random(1, 6) | nothing())
        m.train(random())
        device = random_device()
        m.to(device)
        x = random_pytorch_tensor().to(device)
        y = m(x)
        return y

这样纯写 pytorch 代码的方式来自动测试

Signed-off-by: daquexian <daquexian566@gmail.com>

daquexian · 2021-07-16T07:58:08Z

oneflow/python/test/modules/automated_test_util.py

+    return flow.tensor(torch_tensor.cpu().numpy())
+
+
+def convert_torch_object_to_flow(x):


原先的每个 generator 生成一份 oneflow 数据和一份 pytorch 数据的行为顺手改掉了。现在每个 generator 只生成 pytorch 数据，再通过 convert_torch_object_to_flow 转成 oneflow 数据

daquexian · 2021-07-16T07:59:50Z

oneflow/python/test/modules/automated_test_util.py

+    def value(self):
+        if self._value is None:
+            self._value = self._calc_value()
+        return self._value


每个 generator 对象会缓存自己本次生成的值，用于 x0 = random(1, 6); x1 = x0 + 1; x2 = x0 + 1 这种情况，计算 x1 和 x2 时 x0 只会被计算一次，x1 和 x2 的值会相等

daquexian · 2021-07-16T08:00:22Z

oneflow/python/test/modules/automated_test_util.py

+        return self._value
+
+    def size(self):
+        return 1


这个 size 用来处理 a | b | c 这样连续的 |，期望的行为是 a b c 各有 1/3 概率

daquexian · 2021-07-16T08:00:57Z

oneflow/python/test/modules/automated_test_util.py

-def random(low, high):
-    def generator(annotation):
+class oneof(generator):
+    def __init__(self, *args, possibility=None):


这个 possiblity 只能通过 keyword 形式传进来，需要注意

daquexian · 2021-07-16T08:02:22Z

oneflow/python/test/modules/automated_test_util.py

-                t = [generator(x) for x in annotation.__args__]
-                return zip(*t)
+                return self._generate(x)
+            if annotation.__origin__ is Tuple or annotation.__origin__ is py_tuple:


tuple 被劫持成了 generator dsl 里的一个操作（这个目前不会暴露到文件外），所以原来 python 的 tuple 现在用 py_tuple 表示

daquexian · 2021-07-16T08:04:08Z

oneflow/python/test/modules/automated_test_util.py

-                    continue
-            flow_data, torch_data = generate(name)
+
+        generator_tuple = tuple(


这里创建一个容纳所有 generator 的新 generator，这样才能让 x0 = random(1, 6); x1 = x0 + 1; x2 = x0 + 1 这种情况计算 x1 和 x2 时 x0 只会被计算一次

oneflow/python/test/modules/test_flatten.py

jackalcooper · 2021-07-16T08:14:47Z

是不是可以把这些介绍写在某个 README 里面，然后出错的时候提示去看这个readme

Signed-off-by: daquexian <daquexian566@gmail.com>

MARD1NO · 2021-07-20T05:33:33Z

oneflow/python/test/modules/automated_test_util/torch_flow_dual_object.py

+
+counter = 0
+
+def GetDualObject(name, pytorch, oneflow):


没看太懂这部分 DualObject的处理，大老师可以简单讲下吗

应该是为了防止多次call method的时候前一次的args被以引用的形式进行了修改，导致后一次call method时接受的args值不正确，所以在上下文中动态generate一个method出来，使得每次call method时args的引用不同

BBuf · 2021-07-20T06:54:01Z

oneflow/python/test/modules/automated_test_util/torch_flow_dual_object.py

+    return np.allclose(torch_tensor.detach().cpu().numpy(), flow_tensor.numpy())
+
+
+def autotest(n=20, auto_backward=True):


待加rtol和atol

BBuf · 2021-07-20T07:00:24Z

oneflow/python/test/modules/automated_test_util/torch_flow_dual_object.py

+def check_tensor_equality(torch_tensor, flow_tensor):
+    # TODO: check dtype
+    if torch_tensor.grad is not None:
+        assert flow_tensor.grad is not None, "OneFlow tensor doesn't have grad while PyTorch tensor has one"


这里可以加一下更明显的错误提示吗，比如输出一下出错的这组随机测试的输入和attr，这样CI有异常的话就可以快速定位错误了

这个比想象中复杂一些，之后再加吧

Signed-off-by: daquexian <daquexian566@gmail.com>

github-actions · 2021-07-21T10:00:53Z

CI failed, removing label automerge

terryzhao127 · 2021-07-21T12:15:40Z

oneflow/python/test_utils/automated_test_util/generators.py

+    def register_flow_to_flow_converter(func):
+        annotation2torch_to_flow_converter[annotation] = func
+        return func
+
+    return register_flow_to_flow_converter


这个register_flow_to_flow_converter名字写错了

没有太get到，拼写问题吗

应该是torch_to_flow吧

terryzhao127 · 2021-07-21T12:43:29Z

oneflow/python/test_utils/automated_test_util/torch_flow_dual_object.py

+def torch_tensor_to_flow(x):
+    return flow.tensor(x.cpu().numpy())


没用上在generators.py中定义的tensor_converter

terryzhao127 · 2021-07-21T12:54:32Z

oneflow/python/test_utils/automated_test_util/torch_flow_dual_object.py

+                            oneflow_args,
+                            oneflow_kwargs,
+                        ) = get_args(pytorch_method, *args, **kwargs)
+                        pytorch_res = pytorch_method(*pytorch_args, **pytorch_kwargs)


这一行多余了

好的，已删除

github-actions · 2021-07-21T13:14:26Z

CI failed, removing label automerge

github-actions · 2021-07-21T15:49:44Z

Speed stats:

GPU Name: GeForce GTX 1080 

PyTorch resnet50 time: 139.1ms (= 4173.0ms / 30, input_shape=[16, 3, 224, 224], backward is enabled)
OneFlow resnet50 time: 125.9ms (= 3776.0ms / 30, input_shape=[16, 3, 224, 224], backward is enabled)
Relative speed: 1.11 (= 139.1ms / 125.9ms)

PyTorch resnet50 time: 84.9ms (= 2546.3ms / 30, input_shape=[8, 3, 224, 224], backward is enabled)
OneFlow resnet50 time: 73.6ms (= 2206.6ms / 30, input_shape=[8, 3, 224, 224], backward is enabled)
Relative speed: 1.15 (= 84.9ms / 73.6ms)

PyTorch resnet50 time: 59.5ms (= 1783.9ms / 30, input_shape=[4, 3, 224, 224], backward is enabled)
OneFlow resnet50 time: 49.0ms (= 1469.1ms / 30, input_shape=[4, 3, 224, 224], backward is enabled)
Relative speed: 1.21 (= 59.5ms / 49.0ms)

PyTorch resnet50 time: 48.6ms (= 1459.3ms / 30, input_shape=[2, 3, 224, 224], backward is enabled)
OneFlow resnet50 time: 46.4ms (= 1390.8ms / 30, input_shape=[2, 3, 224, 224], backward is enabled)
Relative speed: 1.05 (= 48.6ms / 46.4ms)

PyTorch resnet50 time: 49.8ms (= 1494.2ms / 30, input_shape=[1, 3, 224, 224], backward is enabled)
OneFlow resnet50 time: 45.3ms (= 1359.8ms / 30, input_shape=[1, 3, 224, 224], backward is enabled)
Relative speed: 1.10 (= 49.8ms / 45.3ms)

terryzhao127 · 2021-07-22T12:41:08Z

oneflow/python/test_utils/automated_test_util/torch_flow_dual_object.py

+                            oneflow_res = torch_tensor_to_flow(pytorch_res)
+                        else:
+                            oneflow_res = oneflow(*oneflow_args, **oneflow_kwargs)
+                        return GetDualObject("unused", pytorch_res, oneflow_res)


这里是不是应该用GetDualObject参数中传入的name啊

terryzhao127 · 2021-07-22T13:31:42Z

oneflow/python/test_utils/automated_test_util/generators.py

+
+@data_generator(torch.Tensor)
+class random_tensor(generator):
+    def __init__(self, ndim=None, dim0=1, dim1=None, dim2=None, dim3=None, dim4=None):


这里的dim0=1是不是写错了

daquexian added 3 commits July 16, 2021 15:33

update automated test framework

ba8b344

Signed-off-by: daquexian <daquexian566@gmail.com>

rename func -> generator

98a72fe

Signed-off-by: daquexian <daquexian566@gmail.com>

reformat

490bf2c

Signed-off-by: daquexian <daquexian566@gmail.com>

daquexian commented Jul 16, 2021

View reviewed changes

daquexian added 8 commits July 16, 2021 17:17

update more powerful conv test

c204156

Signed-off-by: daquexian <daquexian566@gmail.com>

new imperative autotest api

9ae0180

Signed-off-by: daquexian <daquexian566@gmail.com>

Merge remote-tracking branch 'origin/master' into update_autotest

5dcc91f

update automated test framework

03ddb17

Signed-off-by: daquexian <daquexian566@gmail.com>

add some tests and refine code

d864f49

Signed-off-by: daquexian <daquexian566@gmail.com>

Merge remote-tracking branch 'origin/master' into update_autotest

5f0c1ba

update test_matmul.py

f2a04ac

Signed-off-by: daquexian <daquexian566@gmail.com>

fix test_matmul.py

1d6c9f4

Signed-off-by: daquexian <daquexian566@gmail.com>

MARD1NO reviewed Jul 20, 2021

View reviewed changes

BBuf reviewed Jul 20, 2021

View reviewed changes

add atol and rtol, fix test_matmul.py

dc58ec7

Signed-off-by: daquexian <daquexian566@gmail.com>

daquexian added ci feature python labels Jul 20, 2021

daquexian requested a review from oneflow-ci-bot July 20, 2021 10:30

BBuf approved these changes Jul 20, 2021

View reviewed changes

oneflow-ci-bot and others added 2 commits July 20, 2021 12:37

auto format by CI

aa29fbf

Merge branch 'master' into update_autotest

da88edb

BBuf added the automerge label Jul 21, 2021

BBuf requested review from oneflow-ci-bot and removed request for oneflow-ci-bot July 21, 2021 01:30

BBuf mentioned this pull request Jul 21, 2021

基于新版自动测试框架重构Module单元测试 #5549

Closed

oneflow-ci-bot removed their request for review July 21, 2021 02:49

reformat

f1c51c3

Signed-off-by: daquexian <daquexian566@gmail.com>

BBuf requested a review from oneflow-ci-bot July 21, 2021 09:15

github-actions bot removed the automerge label Jul 21, 2021

oneflow-ci-bot removed their request for review July 21, 2021 10:16

terryzhao127 reviewed Jul 21, 2021

View reviewed changes

jackalcooper added 2 commits July 21, 2021 20:23

dirtyfix

1070956

fixname

fab8cf6

BBuf requested a review from oneflow-ci-bot July 21, 2021 12:26

BBuf added the automerge label Jul 21, 2021

fix

00a5d6d

terryzhao127 reviewed Jul 21, 2021

View reviewed changes

Merge branch 'master' into update_autotest

2c41fc4

oneflow-ci-bot requested review from oneflow-ci-bot and removed request for oneflow-ci-bot July 21, 2021 12:45

terryzhao127 reviewed Jul 21, 2021

View reviewed changes

github-actions bot removed the automerge label Jul 21, 2021

BBuf added 2 commits July 21, 2021 21:21

fix bug

ece51c8

fix comments

2698e14

BBuf added the automerge label Jul 21, 2021

oneflow-ci-bot removed their request for review July 21, 2021 15:12

Merge branch 'master' into update_autotest

a10519e

oneflow-ci-bot self-requested a review July 21, 2021 15:12

oneflow-ci-bot merged commit 865ae1a into master Jul 21, 2021

oneflow-ci-bot deleted the update_autotest branch July 21, 2021 16:20

oneflow-ci-bot removed their request for review July 21, 2021 16:20

terryzhao127 reviewed Jul 22, 2021

View reviewed changes

zzk0 mentioned this pull request Oct 18, 2021

[new op] CELU #6544

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update autotest framework #5520

Update autotest framework #5520

daquexian commented Jul 16, 2021 •

edited

Loading

daquexian Jul 16, 2021

daquexian Jul 16, 2021

daquexian Jul 16, 2021

daquexian Jul 16, 2021

daquexian Jul 16, 2021

daquexian Jul 16, 2021

jackalcooper commented Jul 16, 2021 •

edited

Loading

MARD1NO Jul 20, 2021

tea321000 Jul 20, 2021

BBuf Jul 20, 2021

daquexian Jul 20, 2021

BBuf Jul 20, 2021

daquexian Jul 20, 2021

github-actions bot commented Jul 21, 2021

terryzhao127 Jul 21, 2021

BBuf Jul 21, 2021

terryzhao127 Jul 21, 2021

terryzhao127 Jul 21, 2021

terryzhao127 Jul 21, 2021

BBuf Jul 21, 2021

github-actions bot commented Jul 21, 2021

github-actions bot commented Jul 21, 2021

terryzhao127 Jul 22, 2021

terryzhao127 Jul 22, 2021

		return flow.tensor(torch_tensor.cpu().numpy())


		def convert_torch_object_to_flow(x):

		return np.allclose(torch_tensor.detach().cpu().numpy(), flow_tensor.numpy())


		def autotest(n=20, auto_backward=True):

		def torch_tensor_to_flow(x):
		return flow.tensor(x.cpu().numpy())

Update autotest framework #5520

Update autotest framework #5520

Conversation

daquexian commented Jul 16, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jackalcooper commented Jul 16, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Jul 21, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Jul 21, 2021

github-actions bot commented Jul 21, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

daquexian commented Jul 16, 2021 •

edited

Loading

jackalcooper commented Jul 16, 2021 •

edited

Loading