Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Related to some test_onnx_backend.py test cases #744

Closed
chrishkchris opened this issue Jun 19, 2020 · 4 comments
Closed

Related to some test_onnx_backend.py test cases #744

chrishkchris opened this issue Jun 19, 2020 · 4 comments

Comments

@chrishkchris
Copy link
Contributor

chrishkchris commented Jun 19, 2020 •
edited
Loading

Some updates:
@joddiy is fixing some problem in onnx

root@d05828f767ee:~/dcsysh/singa/test/python# python3 test_onnx_backend.py
ss............................ssssssssssssssssssssssssssssssss................ssss..ss..ss......FFFF..ssssssssss..ssssssssssssssss............ssss....................ssssssss........................ssss....ssssssssssssssssssssssssss........ssssssssssssssssssssssssssss..........ssssssssssssss......FsFsssssssssssssssss..................ssss....ssssss..ssss..........ss.s............ssss....ssssssssssssssssssss........ssssssssssss..............ssssssssssssssssssssssss......ss......ssss..ss........FFFF..ssssssssss............ssssssssssssssssssssssssss......ssss....ssssssssssssssssss..................................ss........ssssssssssssssss....ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss................ssssssssssssssssssssssssssssssss................ssssssssssssssssss....................ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss..........ssss......ssss........ssss..............ss..........................................ssssssssssss....................ssssssssssssssssssss....ssssssssssssssssssssssssssssss................ssssss................
======================================================================
FAIL: test_averagepool_2d_same_lower_cpu (__main__.OnnxBackendNodeModelTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line 248, in device_test_func
    return test_func(*args, device=device, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line 313, in run
    atol=model_test.atol)
  File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line 178, in assert_similar_outputs
    atol=atol)
  File "/usr/local/lib/python3.6/dist-packages/numpy/testing/_private/utils.py", line 1533, in assert_allclose
    verbose=verbose, header=header, equal_nan=equal_nan)
  File "/usr/local/lib/python3.6/dist-packages/numpy/testing/_private/utils.py", line 846, in assert_array_compare
    raise AssertionError(msg)
AssertionError:
Not equal to tolerance rtol=0.001, atol=1e-07

Mismatched elements: 189 / 3072 (6.15%)
Max absolute difference: 1.3230393
Max relative difference: 3.
 x: array([[[[ 1.764052e+00,  1.082105e+00,  6.894476e-01, ...,
           1.501069e+00,  8.121531e-01,  2.665550e-01],
         [ 4.381333e-01, -1.760931e-01, -2.374533e-01, ...,...
 y: array([[[[ 4.410131e-01,  5.410524e-01,  3.447238e-01, ...,
           7.505345e-01,  4.060766e-01,  1.332775e-01],
         [ 2.190667e-01, -1.760931e-01, -2.374533e-01, ...,...

======================================================================
FAIL: test_averagepool_2d_same_lower_cuda (__main__.OnnxBackendNodeModelTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line 248, in device_test_func
    return test_func(*args, device=device, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line 313, in run
    atol=model_test.atol)
  File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line 178, in assert_similar_outputs
    atol=atol)
  File "/usr/local/lib/python3.6/dist-packages/numpy/testing/_private/utils.py", line 1533, in assert_allclose
    verbose=verbose, header=header, equal_nan=equal_nan)
  File "/usr/local/lib/python3.6/dist-packages/numpy/testing/_private/utils.py", line 846, in assert_array_compare
    raise AssertionError(msg)
AssertionError:
Not equal to tolerance rtol=0.001, atol=1e-07

Mismatched elements: 189 / 3072 (6.15%)
Max absolute difference: 1.3230393
Max relative difference: 3.
 x: array([[[[ 1.764052e+00,  1.082105e+00,  6.894476e-01, ...,
           1.501069e+00,  8.121531e-01,  2.665550e-01],
         [ 4.381333e-01, -1.760931e-01, -2.374533e-01, ...,...
 y: array([[[[ 4.410131e-01,  5.410524e-01,  3.447238e-01, ...,
           7.505345e-01,  4.060766e-01,  1.332775e-01],
         [ 2.190667e-01, -1.760931e-01, -2.374533e-01, ...,...

======================================================================
FAIL: test_averagepool_2d_same_upper_cpu (__main__.OnnxBackendNodeModelTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line 248, in device_test_func
    return test_func(*args, device=device, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line 313, in run
    atol=model_test.atol)
  File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line 178, in assert_similar_outputs
    atol=atol)
  File "/usr/local/lib/python3.6/dist-packages/numpy/testing/_private/utils.py", line 1533, in assert_allclose
    verbose=verbose, header=header, equal_nan=equal_nan)
  File "/usr/local/lib/python3.6/dist-packages/numpy/testing/_private/utils.py", line 846, in assert_array_compare
    raise AssertionError(msg)
AssertionError:
Not equal to tolerance rtol=0.001, atol=1e-07

Mismatched elements: 189 / 3072 (6.15%)
Max absolute difference: 0.9547153
Max relative difference: 3.
 x: array([[[[-0.176093, -0.237453,  0.757017, ...,  0.112902, -0.50158 ,
          -0.67406 ],
         [-0.773234, -1.090172, -0.339745, ...,  0.040076, -0.369122,...
 y: array([[[[-0.176093, -0.237453,  0.757017, ...,  0.112902, -0.50158 ,
          -0.33703 ],
         [-0.773234, -1.090172, -0.339745, ...,  0.040076, -0.369122,...

======================================================================
FAIL: test_averagepool_2d_same_upper_cuda (__main__.OnnxBackendNodeModelTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line 248, in device_test_func
    return test_func(*args, device=device, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line 313, in run
    atol=model_test.atol)
  File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line 178, in assert_similar_outputs
    atol=atol)
  File "/usr/local/lib/python3.6/dist-packages/numpy/testing/_private/utils.py", line 1533, in assert_allclose
    verbose=verbose, header=header, equal_nan=equal_nan)
  File "/usr/local/lib/python3.6/dist-packages/numpy/testing/_private/utils.py", line 846, in assert_array_compare
    raise AssertionError(msg)
AssertionError:
Not equal to tolerance rtol=0.001, atol=1e-07

Mismatched elements: 189 / 3072 (6.15%)
Max absolute difference: 0.9547153
Max relative difference: 3.
 x: array([[[[-0.176093, -0.237453,  0.757017, ...,  0.112902, -0.50158 ,
          -0.67406 ],
         [-0.773234, -1.090172, -0.339745, ...,  0.040076, -0.369122,...
 y: array([[[[-0.176093, -0.237453,  0.757017, ...,  0.112902, -0.50158 ,
          -0.33703 ],
         [-0.773234, -1.090172, -0.339745, ...,  0.040076, -0.369122,...

======================================================================
FAIL: test_equal_bcast_cpu (__main__.OnnxBackendNodeModelTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line 248, in device_test_func
    return test_func(*args, device=device, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line 313, in run
    atol=model_test.atol)
  File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line 178, in assert_similar_outputs
    atol=atol)
  File "/usr/local/lib/python3.6/dist-packages/numpy/testing/_private/utils.py", line 1533, in assert_allclose
    verbose=verbose, header=header, equal_nan=equal_nan)
  File "/usr/local/lib/python3.6/dist-packages/numpy/testing/_private/utils.py", line 846, in assert_array_compare
    raise AssertionError(msg)
AssertionError:
Not equal to tolerance rtol=0.001, atol=1e-07

Mismatched elements: 1 / 60 (1.67%)
 x: array([[[False, False, False, False, False],
        [False, False, False, False, False],
        [False, False, False, False, False],...
 y: array([[[False, False, False, False, False],
        [False, False, False, False, False],
        [False, False, False, False, False],...

======================================================================
FAIL: test_equal_cpu (__main__.OnnxBackendNodeModelTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line 248, in device_test_func
    return test_func(*args, device=device, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line 313, in run
    atol=model_test.atol)
  File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line 178, in assert_similar_outputs
    atol=atol)
  File "/usr/local/lib/python3.6/dist-packages/numpy/testing/_private/utils.py", line 1533, in assert_allclose
    verbose=verbose, header=header, equal_nan=equal_nan)
  File "/usr/local/lib/python3.6/dist-packages/numpy/testing/_private/utils.py", line 846, in assert_array_compare
    raise AssertionError(msg)
AssertionError:
Not equal to tolerance rtol=0.001, atol=1e-07

Mismatched elements: 1 / 60 (1.67%)
 x: array([[[False, False, False, False, False],
        [False, False, False, False, False],
        [False, False, False, False,  True],...
 y: array([[[False, False, False, False, False],
        [False, False, False, False, False],
        [False, False, False, False,  True],...

======================================================================
FAIL: test_maxpool_2d_same_lower_cpu (__main__.OnnxBackendNodeModelTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line 248, in device_test_func
    return test_func(*args, device=device, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line 313, in run
    atol=model_test.atol)
  File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line 178, in assert_similar_outputs
    atol=atol)
  File "/usr/local/lib/python3.6/dist-packages/numpy/testing/_private/utils.py", line 1533, in assert_allclose
    verbose=verbose, header=header, equal_nan=equal_nan)
  File "/usr/local/lib/python3.6/dist-packages/numpy/testing/_private/utils.py", line 846, in assert_array_compare
    raise AssertionError(msg)
AssertionError:
Not equal to tolerance rtol=0.001, atol=1e-07

Mismatched elements: 35 / 3072 (1.14%)
Max absolute difference: 1.6961312
Max relative difference: 0.
 x: array([[[[ 1.764052e+00,  1.764052e+00,  9.787380e-01, ...,
           1.532779e+00,  1.469359e+00,  3.781625e-01],
         [ 1.764052e+00,  1.764052e+00,  9.787380e-01, ...,...
 y: array([[[[ 1.764052e+00,  1.764052e+00,  9.787380e-01, ...,
           1.532779e+00,  1.469359e+00,  3.781625e-01],
         [ 1.764052e+00,  1.764052e+00,  9.787380e-01, ...,...

======================================================================
FAIL: test_maxpool_2d_same_lower_cuda (__main__.OnnxBackendNodeModelTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line 248, in device_test_func
    return test_func(*args, device=device, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line 313, in run
    atol=model_test.atol)
  File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line 178, in assert_similar_outputs
    atol=atol)
  File "/usr/local/lib/python3.6/dist-packages/numpy/testing/_private/utils.py", line 1533, in assert_allclose
    verbose=verbose, header=header, equal_nan=equal_nan)
  File "/usr/local/lib/python3.6/dist-packages/numpy/testing/_private/utils.py", line 846, in assert_array_compare
    raise AssertionError(msg)
AssertionError:
Not equal to tolerance rtol=0.001, atol=1e-07

Mismatched elements: 35 / 3072 (1.14%)
Max absolute difference: 1.6961312
Max relative difference: 0.
 x: array([[[[ 1.764052e+00,  1.764052e+00,  9.787380e-01, ...,
           1.532779e+00,  1.469359e+00,  3.781625e-01],
         [ 1.764052e+00,  1.764052e+00,  9.787380e-01, ...,...
 y: array([[[[ 1.764052e+00,  1.764052e+00,  9.787380e-01, ...,
           1.532779e+00,  1.469359e+00,  3.781625e-01],
         [ 1.764052e+00,  1.764052e+00,  9.787380e-01, ...,...

======================================================================
FAIL: test_maxpool_2d_same_upper_cpu (__main__.OnnxBackendNodeModelTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line 248, in device_test_func
    return test_func(*args, device=device, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line 313, in run
    atol=model_test.atol)
  File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line 178, in assert_similar_outputs
    atol=atol)
  File "/usr/local/lib/python3.6/dist-packages/numpy/testing/_private/utils.py", line 1533, in assert_allclose
    verbose=verbose, header=header, equal_nan=equal_nan)
  File "/usr/local/lib/python3.6/dist-packages/numpy/testing/_private/utils.py", line 846, in assert_array_compare
    raise AssertionError(msg)
AssertionError:
Not equal to tolerance rtol=0.001, atol=1e-07

Mismatched elements: 37 / 3072 (1.2%)
Max absolute difference: 1.2028884
Max relative difference: 0.
 x: array([[[[ 1.764052,  0.978738,  2.240893, ...,  1.469359,  0.378163,
           0.378163],
         [ 0.177426, -0.347912,  0.462782, ...,  0.976639,  0.706573,...
 y: array([[[[ 1.764052,  0.978738,  2.240893, ...,  1.469359,  0.378163,
           0.378163],
         [ 0.177426, -0.347912,  0.462782, ...,  0.976639,  0.706573,...

======================================================================
FAIL: test_maxpool_2d_same_upper_cuda (__main__.OnnxBackendNodeModelTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line 248, in device_test_func
    return test_func(*args, device=device, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line 313, in run
    atol=model_test.atol)
  File "/usr/local/lib/python3.6/dist-packages/onnx/backend/test/runner/__init__.py", line 178, in assert_similar_outputs
    atol=atol)
  File "/usr/local/lib/python3.6/dist-packages/numpy/testing/_private/utils.py", line 1533, in assert_allclose
    verbose=verbose, header=header, equal_nan=equal_nan)
  File "/usr/local/lib/python3.6/dist-packages/numpy/testing/_private/utils.py", line 846, in assert_array_compare
    raise AssertionError(msg)
AssertionError:
Not equal to tolerance rtol=0.001, atol=1e-07

Mismatched elements: 37 / 3072 (1.2%)
Max absolute difference: 1.2028884
Max relative difference: 0.
 x: array([[[[ 1.764052,  0.978738,  2.240893, ...,  1.469359,  0.378163,
           0.378163],
         [ 0.177426, -0.347912,  0.462782, ...,  0.976639,  0.706573,...
 y: array([[[[ 1.764052,  0.978738,  2.240893, ...,  1.469359,  0.378163,
           0.378163],
         [ 0.177426, -0.347912,  0.462782, ...,  0.976639,  0.706573,...

----------------------------------------------------------------------
Ran 1114 tests in 2.126s
@chrishkchris

This comment has been minimized.

Sign in to view
@chrishkchris chrishkchris changed the title Could we override the test precision atot of onnx test case, atot=1e-7 is too tough? Could we override the test precision atot of onnx test case, atol=1e-7 is too tough? Jun 19, 2020
@joddiy
Copy link
Member

joddiy commented Jun 19, 2020 •
edited
Loading

@chrishkchris I have fixed the average pool and max pool issues above in #746

However, for the equal, I found this is because there is some error when we do the sub operator for int32 input:

for example for input:

A = np.array([[[17, 4, 9, 22, 18], [-9, 9, -1, -1, 4], [1, 14, 7, 1, 4],
                [3, 14, -2, 3, -8]],
                [[-25, 6, 8, -7, 22], [-14, 0, -1, 15, 14],
                [1, 3, -8, -19, -3], [1, 12, 12, -3, -3]],
                [[-10, -14, -17, 19, -5], [-4, -12, 7, -16, -2],
                [-8, 3, -5, -11, 0], [4, 0, 3, -6, -3]]],
                    dtype=np.int32)
B = np.array([[[-6, -3, -8, -17, 1], [-4, -16, 4, -9, 0],
                [7, 1, 11, -12, 4], [-6, -8, -5, -3, 0]],
                [[-11, 9, 4, -15, 14], [18, 11, -1, -10, 10],
                [-4, 12, 2, 9, 3], [7, 0, 17, 1, 4]],
                [[18, -13, -12, 9, -11], [19, -4, -7, 19, 14],
                [18, 9, -8, 19, -2], [8, 9, -1, 6, 9]]],
                    dtype=np.int32)

The result of singa.__sub__(a, b) is:

[[[         -6          -3          -8         -17          17]
  [         -9          -3          -1          -1           3]
  [         -6          -3          -8         -17           3]
  [         -6          -3          -2         -17          -8]]

 [[        -25          -3          -8          -7          21]
  [        -14          -3          -1         -17          13]
  [         -6          -3          -8         -19          -3]
  [         -6          -3          -8          -3          -3]]

 [[        -10         -14         -17         -17          -5]
  [         -4         -12          -8         -16          -2]
  [         -8          -3          -5         -11 -2147483647]
  [         -6          -3          -8          -6          -3]]]
Fs[[[         -6          -3          -8         -17          17]
  [         -9         -16          -1          -1           4]
  [-2147483642          13 -2147483644         -12           0]
  [         -6          -8          -2          -3          -8]]

 [[        -25 -2147483645           4          -7           8]
  [        -14 -2147483637          -1         -10           4]
  [         -4 -2147483639          -8         -19          -3]
  [-2147483642          12 -2147483643          -3          -3]]

 [[        -10         -14         -17          10          -5]
  [         -4         -12          -7         -16          -2]
  [         -8 -2147483642          -5         -11          -2]
  [-2147483644 -2147483639          -1          -6          -3]]]

The correct result should be:

[[[ 23,   7,  17,  39,  17],
        [ -5,  25,  -5,   8,   4],
        [ -6,  13,  -4,  13,   0],
        [  9,  22,   3,   6,  -8]],

       [[-14,  -3,   4,   8,   8],
        [-32, -11,   0,  25,   4],
        [  5,  -9, -10, -28,  -6],
        [ -6,  12,  -5,  -4,  -7]],

       [[-28,  -1,  -5,  10,   6],
        [-23,  -8,  14, -35, -16],
        [-26,  -6,   3, -30,   2],
        [ -4,  -9,   4, -12, -12]]]

@dcslin can we fix it?

@chrishkchris chrishkchris changed the title Could we override the test precision atot of onnx test case, atol=1e-7 is too tough? Related to some test_onnx_backend.py test cases Jun 21, 2020
@dcslin
Copy link
Member

dcslin commented Jul 13, 2020

@chrishkchris I have fixed the average pool and max pool issues above in #746

However, for the equal, I found this is because there is some error when we do the sub operator for int32 input:

for example for input:

A = np.array([[[17, 4, 9, 22, 18], [-9, 9, -1, -1, 4], [1, 14, 7, 1, 4],
                [3, 14, -2, 3, -8]],
                [[-25, 6, 8, -7, 22], [-14, 0, -1, 15, 14],
                [1, 3, -8, -19, -3], [1, 12, 12, -3, -3]],
                [[-10, -14, -17, 19, -5], [-4, -12, 7, -16, -2],
                [-8, 3, -5, -11, 0], [4, 0, 3, -6, -3]]],
                    dtype=np.int32)
B = np.array([[[-6, -3, -8, -17, 1], [-4, -16, 4, -9, 0],
                [7, 1, 11, -12, 4], [-6, -8, -5, -3, 0]],
                [[-11, 9, 4, -15, 14], [18, 11, -1, -10, 10],
                [-4, 12, 2, 9, 3], [7, 0, 17, 1, 4]],
                [[18, -13, -12, 9, -11], [19, -4, -7, 19, 14],
                [18, 9, -8, 19, -2], [8, 9, -1, 6, 9]]],
                    dtype=np.int32)

The result of singa.__sub__(a, b) is:

[[[         -6          -3          -8         -17          17]
  [         -9          -3          -1          -1           3]
  [         -6          -3          -8         -17           3]
  [         -6          -3          -2         -17          -8]]

 [[        -25          -3          -8          -7          21]
  [        -14          -3          -1         -17          13]
  [         -6          -3          -8         -19          -3]
  [         -6          -3          -8          -3          -3]]

 [[        -10         -14         -17         -17          -5]
  [         -4         -12          -8         -16          -2]
  [         -8          -3          -5         -11 -2147483647]
  [         -6          -3          -8          -6          -3]]]
Fs[[[         -6          -3          -8         -17          17]
  [         -9         -16          -1          -1           4]
  [-2147483642          13 -2147483644         -12           0]
  [         -6          -8          -2          -3          -8]]

 [[        -25 -2147483645           4          -7           8]
  [        -14 -2147483637          -1         -10           4]
  [         -4 -2147483639          -8         -19          -3]
  [-2147483642          12 -2147483643          -3          -3]]

 [[        -10         -14         -17          10          -5]
  [         -4         -12          -7         -16          -2]
  [         -8 -2147483642          -5         -11          -2]
  [-2147483644 -2147483639          -1          -6          -3]]]

The correct result should be:

[[[ 23,   7,  17,  39,  17],
        [ -5,  25,  -5,   8,   4],
        [ -6,  13,  -4,  13,   0],
        [  9,  22,   3,   6,  -8]],

       [[-14,  -3,   4,   8,   8],
        [-32, -11,   0,  25,   4],
        [  5,  -9, -10, -28,  -6],
        [ -6,  12,  -5,  -4,  -7]],

       [[-28,  -1,  -5,  10,   6],
        [-23,  -8,  14, -35, -16],
        [-26,  -6,   3, -30,   2],
        [ -4,  -9,   4, -12, -12]]]

@dcslin can we fix it?

this int subtraction testing is fixed and added in #763

@chrishkchris chrishkchris mentioned this issue Jul 13, 2020
Merged
@chrishkchris
Copy link
Contributor Author

solved by #763

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@dcslin @joddiy @chrishkchris