Add Large Tensor Test for linalg_syrk #18782

Zha0q1 · 2020-07-23T20:41:46Z

Description

Check if syrk_batch works correctly with large tensors. This test will fail with the current code base; #18752 should fix it.

TODO:

make test function naming consistent with other large tensor tests (need to rebase after any of those tests are merged)
merge this only after the fix has been merged

This test passes on both BLAS int32 and 64 builds.

ubuntu@ip-172-31-43-103:~$ MXNET_TEST_COUNT=10000 nosetests --logging-level=DEBUG --verbose -s mxnet/tests/nightly/test_large_array.py:test_linalg_operators
test_large_array.test_linalg_operators ... [23:14:57] ../src/storage/storage.cc:198: Using Pooled (Naive) StorageManager for CPU
ok

----------------------------------------------------------------------
Ran 1 test in 637.245s

OK

Checklist

Essentials

Please feel free to remove inapplicable items for your PR.

The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to the relevant JIRA issue created (except PRs with tiny changes)
Changes are complete (i.e. I finished coding on this PR)
All changes have test coverage:
Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
Code is well-documented:
For user-facing API changes, API doc string has been updated.
For new C++ functions in header files, their functionalities and arguments are documented.
For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
Check the API doc at https://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
To the best of my knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

Changes

Feature1, tests, (and when applicable, API doc)
Feature2, tests, (and when applicable, API doc)

Comments

If this change is a backward incompatible change, why must this change be made.
Interesting edge cases to note here

mxnet-bot · 2020-07-23T20:41:48Z

Hey @Zha0q1 , Thanks for submitting the PR
All tests are already queued to run once. If tests fail, you can trigger one or more tests again with the following commands:

To trigger all jobs: @mxnet-bot run ci [all]
To trigger specific jobs: @mxnet-bot run ci [job1, job2]

CI supported jobs: [windows-cpu, centos-cpu, windows-gpu, sanity, edge, miscellaneous, clang, centos-gpu, website, unix-gpu, unix-cpu]

Note:
Only following 3 categories can trigger CI :PR Author, MXNet Committer, Jenkins Admin.
All CI tests must pass before the PR can be merged.

access2rohit · 2020-07-23T21:52:47Z

@Zha0q1 can you fix this issue with your PR "This branch is out-of-date with the base branch"

Zha0q1 · 2020-07-23T22:18:59Z

@Zha0q1 can you fix this issue with your PR "This branch is out-of-date with the base branch"

Fixed.

access2rohit · 2020-07-24T15:45:42Z

tests/nightly/test_large_array.py

+        A.attach_grad()
+        with mx.autograd.record():
+            out = nd.linalg.syrk(A, alpha=2, transpose=False)
+        for i in range(LARGE_SQ_X):


You can check in 2 places in (0,y,y) and (1,y,y). No need to check in 70000 locations

access2rohit · 2020-07-24T15:46:10Z

tests/nightly/test_large_array.py

+            assert out[0,i,i] == 2
+            assert_almost_equal(out[1,i,i], nd.array([0.02]), rtol=1e-3, atol=1e-5)
+        out.backward()
+        for i in range(LARGE_SQ_X):


access2rohit · 2020-07-24T15:53:25Z

tests/nightly/test_large_array.py

+        for i in range(LARGE_SQ_X):
+            # check the first row
+            assert A.grad[0,0,i] == 4
+            assert_almost_equal(A.grad[1,0,i], nd.array([0.4]), rtol=1e-3, atol=1e-5)


Question: Why did this become 0.4 and not 0.04 ? OR just let me know if this output is consistent with smaller inputs like 2x2 or 3x3.

This is the correct result I believe. I verified with hand-written calculation. Yeah it also struck as counter-intuitive to me.. I am going to dive deep in matrix grad when I find time

Can you just check with smaller input run and let me know the results. That should be good enough

access2rohit · 2020-07-24T15:59:02Z

Few comments ... overall code is good

access2rohit

Can you just check with smaller input run and let me know the results. That should be good enough .... overall LGTM!

ChaiBapchya

LGTM
Thanks!

Zha0q1 · 2020-07-24T22:42:29Z

Can you just check with smaller input run and let me know the results. That should be good enough .... overall LGTM!

This has been tested with both small and large input tensors

szha · 2020-07-24T22:46:34Z

tests/nightly/test_large_array.py

@@ -37,7 +37,7 @@
 LARGE_X = 100000000
 SMALL_X = 100
 SMALL_Y = 50
-LARGE_SQ_X = 80000
+LARGE_SQ_X = 70000


70000*70000/2**32 is just over 2**32
80k is lot more

We figured 7000 is large enough to overflow int 32. This is a new constant just introduced so the few of us decided to tweak it to 70000

70k * 70k = 4.9 billion
int32 range < 4.3 billion
Therefore increasing input size will only increase test runtime.

ChaiBapchya · 2020-07-24T22:49:19Z

@Zha0q1 also add your name to Contributors.md in the upcoming PR [let's not retrigger CI for that].
https://github.com/apache/incubator-mxnet/blob/master/CONTRIBUTORS.md
Thanks a lot for your contributions! :)

Zha0q1 · 2020-07-24T22:50:18Z

@Zha0q1 also add your name to Contributors.md in the upcoming PR [let's not retrigger CI for that].
https://github.com/apache/incubator-mxnet/blob/master/CONTRIBUTORS.md
Thanks a lot for your contributions! :)

will do!

* add large tensor test for syrk, foward and backward * change to batch input * move syrk test into test-linalg Co-authored-by: Ubuntu <ubuntu@ip-172-31-6-47.us-west-2.compute.internal>

add large tensor test for syrk, foward and backward

6c1a904

Merge branch 'v1.x' into add_syrk_large_tensor_test

0cefe25

change to batch input

ae4318a

access2rohit reviewed Jul 24, 2020

View reviewed changes

Zha0q1 and others added 2 commits July 24, 2020 09:52

Merge branch 'v1.x' into add_syrk_large_tensor_test

8d9b188

move syrk test into test-linalg

b49f999

access2rohit approved these changes Jul 24, 2020

View reviewed changes

ChaiBapchya approved these changes Jul 24, 2020

View reviewed changes

Zha0q1 changed the title ~~[WIP] Add Large Tensor Test for linalg_syrk~~ Add Large Tensor Test for linalg_syrk Jul 24, 2020

szha reviewed Jul 24, 2020

View reviewed changes

szha merged commit 85ff00d into apache:v1.x Jul 24, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Large Tensor Test for linalg_syrk #18782

Add Large Tensor Test for linalg_syrk #18782

Zha0q1 commented Jul 23, 2020

mxnet-bot commented Jul 23, 2020

access2rohit commented Jul 23, 2020

Zha0q1 commented Jul 23, 2020

access2rohit Jul 24, 2020

access2rohit Jul 24, 2020

access2rohit Jul 24, 2020 •

edited

Zha0q1 Jul 24, 2020

access2rohit Jul 24, 2020

access2rohit commented Jul 24, 2020

access2rohit left a comment

ChaiBapchya left a comment

Zha0q1 commented Jul 24, 2020

szha Jul 24, 2020

ChaiBapchya Jul 24, 2020 •

edited

Zha0q1 Jul 24, 2020 •

edited

access2rohit Jul 27, 2020

ChaiBapchya commented Jul 24, 2020 •

edited

Zha0q1 commented Jul 24, 2020

Add Large Tensor Test for linalg_syrk #18782

Add Large Tensor Test for linalg_syrk #18782

Conversation

Zha0q1 commented Jul 23, 2020

Description

Checklist

Essentials

Changes

Comments

mxnet-bot commented Jul 23, 2020

access2rohit commented Jul 23, 2020

Zha0q1 commented Jul 23, 2020

access2rohit Jul 24, 2020

Choose a reason for hiding this comment

access2rohit Jul 24, 2020

Choose a reason for hiding this comment

access2rohit Jul 24, 2020 • edited

Choose a reason for hiding this comment

Zha0q1 Jul 24, 2020

Choose a reason for hiding this comment

access2rohit Jul 24, 2020

Choose a reason for hiding this comment

access2rohit commented Jul 24, 2020

access2rohit left a comment

Choose a reason for hiding this comment

ChaiBapchya left a comment

Choose a reason for hiding this comment

Zha0q1 commented Jul 24, 2020

szha Jul 24, 2020

Choose a reason for hiding this comment

ChaiBapchya Jul 24, 2020 • edited

Choose a reason for hiding this comment

Zha0q1 Jul 24, 2020 • edited

Choose a reason for hiding this comment

access2rohit Jul 27, 2020

Choose a reason for hiding this comment

ChaiBapchya commented Jul 24, 2020 • edited

Zha0q1 commented Jul 24, 2020

access2rohit Jul 24, 2020 •

edited

ChaiBapchya Jul 24, 2020 •

edited

Zha0q1 Jul 24, 2020 •

edited

ChaiBapchya commented Jul 24, 2020 •

edited