Add Jenkins setup for running nightly tests on examples #13270

kalyc · 2018-11-14T21:49:11Z

Description

Add Jenkins setup for running nightly tests on examples

Checklist

Essentials

Please feel free to remove inapplicable items for your PR.

The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to the relevant JIRA issue created (except PRs with tiny changes)
Changes are complete (i.e. I finished coding on this PR)
All changes have test coverage:
Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
Code is well-documented:
For user-facing API changes, API doc string has been updated.
For new C++ functions in header files, their functionalities and arguments are documented.
For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
Check the API doc at http://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

Changes

Add Jenkins setup for running nightly tests on examples for python2 & python3 on GPU

Comments

PR as per discussion offline with @marcoabreu

kalyc · 2018-11-14T21:50:19Z

@marcoabreu @ankkhedia

kalyc · 2018-11-14T21:50:45Z

@mxnet-label-bot add [pr-awaiting-review]

roywei

Thanks for adding the example test!
How do we test this?

roywei · 2018-11-15T00:04:14Z

ci/docker/runtime_functions.sh

+    export MXNET_STORAGE_FALLBACK_LOG_VERBOSE=0
+    export MXNET_TEST_KERNEL=python3
+    cd /work/mxnet/tests/examples
+    nosetests-3.4 --with-xunit --xunit-file nosetests_tutorials.xml test_examples.py --nologcapture


Does this PR depend on #13271 to be merged?

as mentioned here - #13271 (comment), please combine the PRs

marcoabreu · 2018-11-15T00:17:13Z

ci/docker/runtime_functions.sh

@@ -1125,6 +1125,30 @@ nightly_tutorial_test_ubuntu_python2_gpu() {
    nosetests-3.4 --with-xunit --xunit-file nosetests_tutorials.xml test_tutorials.py --nologcapture
 }

+nightly_example_test_ubuntu_python3_gpu() {


Please review everything in here. It seems like this has just been copy and pasted

Yes we need a similar setup - as we want to run the examples on GPU.
We want to know how to test the jenkins file locally so the code can be tested

ci/docker/runtime_functions.sh

anirudhacharya · 2018-11-15T07:53:09Z

ci/docker/runtime_functions.sh

+
+}
+
+nightly_example_test_ubuntu_python2_gpu() {


do we really need to test examples on python2? From what I understand only newly fixed examples are going to get added to the tests here and these will be python 3 compliant. And python 2 is getting EOLed in a year's time.

We haven't moved off of python2 yet - the other nightly tests also run for python2

ci/docker/runtime_functions.sh

marcoabreu · 2018-11-15T11:11:53Z

@jlcontreras

kalyc · 2018-11-15T17:49:31Z

Addressed comments and removed extraneous commands from jenkins setup

stu1130 · 2018-11-26T17:48:46Z

@marcoabreu @anirudhacharya @ankkhedia looks like your comments were addressed?

marcoabreu · 2018-11-27T16:06:53Z

Did we test this yet?

kalyc · 2018-11-27T18:00:12Z

@marcoabreu we have cut a TT at your end for assistance in helping with testing this setup. Could you help us out with it?

marcoabreu · 2018-11-27T18:29:07Z

@Chancebair will assist you this week Am Di., 27. Nov. 2018, 19:01 hat Kalyanee Chendke <notifications@github.com> geschrieben:

…

@marcoabreu <https://github.com/marcoabreu> we have cut a TT at your end for assistance in helping with testing this setup. Could you help us out with it? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#13270 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ARxB6_KSQis8D8wv81qHIopNWwDcCHkYks5uzX3kgaJpZM4YepVv> .

kalyc · 2018-11-29T18:44:33Z

@Chancebair could you help with testing this PR?

nswamy · 2018-12-08T04:21:02Z

@kalyc could you please follow up and take it to completion.

Roshrini · 2019-01-02T19:39:32Z

@kalyc Thanks for working on this! Is this PR good to go? Did it pass tests on jenkins setup?

kalyc · 2019-01-02T19:41:03Z

@Roshrini looks like there is a build failure with this setup - http://jenkins.mxnet-ci-dev.amazon-ml.com/job/test-kalyc-NightlyTestsForBinaries/job/jenkins_setup_testing_examples/

Requesting @ankkhedia @roywei to take a look. Please let me know if this PR needs to be closed accordingly.

kalyc · 2019-01-02T19:48:36Z

Error log -

======================================================================
FAIL: test_examples.test_cifar_gpu

Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/nose/case.py", line 198, in runTest
self.test(*self.arg)
File "/work/mxnet/tests/examples/test_examples.py", line 79, in test_cifar_gpu
['python', os.path.join(example_dir, 'train_cifar10.py'), '--num-epochs', '5', 'gpus', '0'])
AssertionError

Ran 2 tests in 6024.068s

…10 example

kalyc · 2019-01-08T20:13:50Z

As discussed offline @ankkhedia and @roywei will take this PR forward. Closing this PR for now.

marcoabreu added the pr-awaiting-review PR is waiting for code review label Nov 14, 2018

roywei reviewed Nov 15, 2018

View reviewed changes

marcoabreu reviewed Nov 15, 2018

View reviewed changes

ankkhedia suggested changes Nov 15, 2018

View reviewed changes

ci/docker/runtime_functions.sh Outdated Show resolved Hide resolved

anirudhacharya mentioned this pull request Nov 15, 2018

[Test] [Example] adding tests for examples #13271

Closed

1 task

anirudhacharya reviewed Nov 15, 2018

View reviewed changes

roywei mentioned this pull request Nov 15, 2018

[Example] fix train mnist for inception-bn and resnet #13239

Merged

5 tasks

kalyc force-pushed the jenkins_setup_testing_examples branch from 76c24eb to a86bbdc Compare December 18, 2018 19:45

kalyc added 4 commits December 18, 2018 15:20

Add Jenkins setup for running nightly tests on examples

0cd8467

Add nostest command

ee065cc

Remove extraneous setup commands

9b00936

Add example tests from PR - 13271

1fcc220

kalyc force-pushed the jenkins_setup_testing_examples branch from a86bbdc to 1fcc220 Compare December 18, 2018 23:30

kalyc added 3 commits January 2, 2019 11:54

Update directory config for testing cifar example

8a5a7b6

Update directory

7209369

Remove test_cifar10_gpu as theres no gpu attribute in the train_cifar…

1694458

…10 example

kalyc closed this Jan 8, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Jenkins setup for running nightly tests on examples #13270

Add Jenkins setup for running nightly tests on examples #13270

kalyc commented Nov 14, 2018

kalyc commented Nov 14, 2018

kalyc commented Nov 14, 2018

roywei left a comment

roywei Nov 15, 2018

kalyc Nov 15, 2018

anirudhacharya Nov 15, 2018

marcoabreu Nov 15, 2018

kalyc Nov 15, 2018

anirudhacharya Nov 15, 2018

kalyc Nov 15, 2018

marcoabreu commented Nov 15, 2018

kalyc commented Nov 15, 2018

stu1130 commented Nov 26, 2018

marcoabreu commented Nov 27, 2018

kalyc commented Nov 27, 2018

marcoabreu commented Nov 27, 2018 via email

kalyc commented Nov 29, 2018

nswamy commented Dec 8, 2018

Roshrini commented Jan 2, 2019

kalyc commented Jan 2, 2019

kalyc commented Jan 2, 2019

kalyc commented Jan 8, 2019


		}

		nightly_example_test_ubuntu_python2_gpu() {

Add Jenkins setup for running nightly tests on examples #13270

Add Jenkins setup for running nightly tests on examples #13270

Conversation

kalyc commented Nov 14, 2018

Description

Checklist

Essentials

Changes

Comments

kalyc commented Nov 14, 2018

kalyc commented Nov 14, 2018

roywei left a comment

Choose a reason for hiding this comment

roywei Nov 15, 2018

Choose a reason for hiding this comment

kalyc Nov 15, 2018

Choose a reason for hiding this comment

anirudhacharya Nov 15, 2018

Choose a reason for hiding this comment

marcoabreu Nov 15, 2018

Choose a reason for hiding this comment

kalyc Nov 15, 2018

Choose a reason for hiding this comment

anirudhacharya Nov 15, 2018

Choose a reason for hiding this comment

kalyc Nov 15, 2018

Choose a reason for hiding this comment

marcoabreu commented Nov 15, 2018

kalyc commented Nov 15, 2018

stu1130 commented Nov 26, 2018

marcoabreu commented Nov 27, 2018

kalyc commented Nov 27, 2018

marcoabreu commented Nov 27, 2018 via email

kalyc commented Nov 29, 2018

nswamy commented Dec 8, 2018

Roshrini commented Jan 2, 2019

kalyc commented Jan 2, 2019

kalyc commented Jan 2, 2019

====================================================================== FAIL: test_examples.test_cifar_gpu

kalyc commented Jan 8, 2019

======================================================================
FAIL: test_examples.test_cifar_gpu