removes python path insert of tests folder for examples #9151

rahul003 · 2017-12-20T03:02:45Z

Description

Some examples insert a folder to the python path to fetch a file from tests/ folder. This causes few problems
1. It requires the user to have the whole mxnet repository, with the directory structure intact
2. It means the user can't run the example standalone
3. For distributed training this means we would have to synchronize the whole mxnet directory across all machines.
This is unnecessary and inconvenient for the user.

Checklist

Essentials

Passed code style checking (make lint)
Changes are complete (i.e. I finished coding on this PR)
All changes have test coverage:
Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
Code is well-documented:
For user-facing API changes, API doc string has been updated.
For new C++ functions in header files, their functionalities and arguments are documented.
For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

Changes

Moves get_mnist_pkl, get_mnist_ubyte, get_mnist_iterator from tests folder to mxnet.test_utils
Adds support to functions that generate data iterators for partitioning of datasets so that different workers in distributed training can use different parts of the dataset.

Signed-off-by: Rahul <rahulhuilgol@gmail.com>

and removing pythonpath inserts for get_data, by moving them to test_utils Signed-off-by: Rahul <rahulhuilgol@gmail.com>

Signed-off-by: Rahul <rahulhuilgol@gmail.com>

Punctuation and minor changes

and address some review comments Signed-off-by: Rahul <rahulhuilgol@gmail.com>

indentation change

cmake instruction

aaronmarkham

Instead of .test_utils can we use gluon model zoo and gluon's data iterator?

eric-haibin-lin · 2017-12-21T22:53:30Z

python/mxnet/test_utils.py

+            zf.extractall('data')
+
+def get_mnist_iterator(batch_size, input_shape, num_parts=1, part_index=0):
+    """Returns training and validation iterators for MNIST dataset


Can we add more documentation to this function about the expected location of the dataset (data/train-images-idx3-ubyte)?

* initial draft * WIP tutorial * WIP tutorial, with mnist script changes Signed-off-by: Rahul <rahulhuilgol@gmail.com> * WIP tutorial, with mnist script changes Signed-off-by: Rahul <rahulhuilgol@gmail.com> * use logger Signed-off-by: Rahul <rahulhuilgol@gmail.com> * remove from old page Signed-off-by: Rahul <rahulhuilgol@gmail.com> * first draft of tutorial and removing pythonpath inserts for get_data, by moving them to test_utils Signed-off-by: Rahul <rahulhuilgol@gmail.com> * fix typos Signed-off-by: Rahul <rahulhuilgol@gmail.com> * rename functions Signed-off-by: Rahul <rahulhuilgol@gmail.com> * small change in section heading Signed-off-by: Rahul <rahulhuilgol@gmail.com> * fix reimport Signed-off-by: Rahul <rahulhuilgol@gmail.com> * Update distributed_training.md * Update distributed_training.md Punctuation and minor changes * fix gluon iterators and address some review comments Signed-off-by: Rahul <rahulhuilgol@gmail.com> * Update multi_devices.md * Update distributed_training.md indentation change * Update distributed_training.md cmake instruction * undo doc changes

rahul003 added 19 commits December 14, 2017 16:25

initial draft

2d64cce

WIP tutorial

0f586c3

WIP tutorial, with mnist script changes

086c968

Signed-off-by: Rahul <rahulhuilgol@gmail.com>

WIP tutorial, with mnist script changes

c18c0f1

Signed-off-by: Rahul <rahulhuilgol@gmail.com>

use logger

efe6b5f

Signed-off-by: Rahul <rahulhuilgol@gmail.com>

remove from old page

d3adec8

Signed-off-by: Rahul <rahulhuilgol@gmail.com>

first draft of tutorial

7c0fdd1

and removing pythonpath inserts for get_data, by moving them to test_utils Signed-off-by: Rahul <rahulhuilgol@gmail.com>

fix typos

d60487d

Signed-off-by: Rahul <rahulhuilgol@gmail.com>

rename functions

6c4e88c

Signed-off-by: Rahul <rahulhuilgol@gmail.com>

fix paths

242417c

small change in section heading

be71a71

Signed-off-by: Rahul <rahulhuilgol@gmail.com>

fix reimport

af97eca

Signed-off-by: Rahul <rahulhuilgol@gmail.com>

Update distributed_training.md

189aca3

Update distributed_training.md

6dbec1f

Punctuation and minor changes

fix gluon iterators

194211d

and address some review comments Signed-off-by: Rahul <rahulhuilgol@gmail.com>

Update multi_devices.md

9298a83

Update distributed_training.md

25f980f

indentation change

Update distributed_training.md

727bdb5

cmake instruction

undo doc changes

40ee98e

This was referenced Dec 20, 2017

[MXNET-37] tutorial for distributed training #9152

Merged

[OLD] tutorial for distributed training #9135

Closed

aaronmarkham reviewed Dec 20, 2017

View reviewed changes

eric-haibin-lin reviewed Dec 21, 2017

View reviewed changes

piiswrong merged commit 11cb609 into apache:master Jan 15, 2018

rahul003 deleted the rm-pythonpath branch April 6, 2018 08:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

removes python path insert of tests folder for examples #9151

removes python path insert of tests folder for examples #9151

rahul003 commented Dec 20, 2017

aaronmarkham left a comment

eric-haibin-lin Dec 21, 2017

removes python path insert of tests folder for examples #9151

removes python path insert of tests folder for examples #9151

Conversation

rahul003 commented Dec 20, 2017

Description

Checklist

Essentials

Changes

aaronmarkham left a comment

Choose a reason for hiding this comment

eric-haibin-lin Dec 21, 2017

Choose a reason for hiding this comment