[MXNET-423] Gluon Model Zoo Pre Trained Model tutorial #10959

ThomasDelteil · 2018-05-15T22:06:13Z

Description

There has been several instances of beginner mxnet users who haven't been able to use the models from the model zoo as they were lacking a fully integrated example. This puts all parts together. There was a blog post recently posted here: https://aws.amazon.com/fr/blogs/machine-learning/use-pre-trained-models-with-apache-mxnet/ using the module API. This is an attempt to draw people towards the nicer Gluon API for pre-trained model.

Checklist

Essentials

Please feel free to remove inapplicable items for your PR.

The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to the relevant JIRA issue created (except PRs with tiny changes)
Changes are complete (i.e. I finished coding on this PR)
All changes have test coverage:
Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
Code is well-documented:
For user-facing API changes, API doc string has been updated.
For new C++ functions in header files, their functionalities and arguments are documented.
For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
Check the API doc at http://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

piiswrong · 2018-05-15T22:09:42Z

BTW where is the entry point for these tutorials? Are they listed on the tutorial page?

ThomasDelteil · 2018-05-15T22:10:31Z

@piiswrong yes the index.md file of the docs/tutorial folder is updated

You can find the tutorials listed here: http://mxnet.incubator.apache.org/tutorials/index.html

ThomasDelteil · 2018-05-15T22:11:28Z

docs/tutorials/index.md

    * [Linear Regression](http://gluon.mxnet.io/chapter02_supervised-learning/linear-regression-gluon.html) <img src="https://upload.wikimedia.org/wikipedia/commons/6/6a/External_link_font_awesome.svg" alt="External link" height="15px" style="margin: 0px 0px 3px 3px;"/>
-    * [MNIST Handwritten Digit Classification](/tutorials/gluon/mnist.html)


this tutorial is already listed three lines above

indhub

Very needed tutorial! Should we talk about freezing parameters during fine-tuning. Or do we want to do that in a different tutorial?

indhub · 2018-05-16T22:17:04Z

docs/tutorials/gluon/pretrained_models.md

+## Loading the model
+
+The [Gluon Model Zoo](https://mxnet.incubator.apache.org/api/python/gluon/model_zoo.html) provides a collection of off-the-shelf models. You can get the ImageNet pre-trained model by using `pretrained=True`. 
+If you want to train on your own classification problem from scratch, you can get an untrained network with a specific number of classes using the `classes=10` for example


using the classes parameter. For example,

indhub · 2018-05-16T22:31:41Z

docs/tutorials/gluon/pretrained_models.md

+filename = mx.test_utils.download('https://github.com/dmlc/web-data/blob/master/mxnet/doc/tutorials/onnx/images/dog.jpg?raw=true', fname='dog.jpg')
+```
+
+If you want to use your own image for the test, upload it in the same folder and change the following line


I assume user has to copy the image to the same folder as the notebook. 'Upload' usually means uploading to a server. Do you want to change this to something along the lines of "copy the image to the same folder that contains the notebook and change the following line"?

indhub · 2018-05-16T22:32:44Z

docs/tutorials/gluon/pretrained_models.md

+
+
+Neural network expects input in a specific format. Usually images comes in the `Width x Height x Channels` format. Where channels are the RGB channels.
+This network accepts images in the `BatchSize x 3 x 224 x 224`. `224 x 224` is the image resolution, that?s how the model was trained. `3` is the number of channels : Red, Green and Blue (in this order). In this case we use a `BatchSize` of `1` since we are predicting one image at a time.


indhub · 2018-05-16T22:35:43Z

docs/tutorials/gluon/pretrained_models.md

+top_pred = predictions.topk(k=3)[0].asnumpy()
+```
+
+And we print the category predicted with the corresponding probability:


categories predicted with their corresponding probabilities.

ThomasDelteil · 2018-05-16T23:20:09Z

@indhub Thanks for the review! I have updated accordingly

I have linked to 2 fine-tuning tutorials which cover quite well the subject, especially the straight dope one since it is using a model from the model zoo as well.

As for freezing the parameters during fine-tuning, my experience has been that it consistently underperformed compared to full network fine-tuning, even with smaller datasets. Do you have any use-case or datasets where freezing parameters during fine-tuning would be advantageous? In my tests, I thought I would at least see faster convergence when freezing all but the last layer, but it took roughly the same time and got worse accuracy than full network training.

safrooze · 2018-05-17T15:42:55Z

Regardless of your personal experience, freezing parameters is a technique used in many papers and I'm sure some users would be looking for a way to experiment with that. I think it makes sense to add it to this tutorial.

ThomasDelteil · 2018-05-17T16:12:36Z

@safrooze I am not disputing the fact that freezing parameters is a useful (and used) technique, especially training difficult to converge networks like GANs where you want to freeze parts of your network at times to help convergence.

However I think it is out of scope of a tutorial called Using pre-trained models from the Gluon Model Zoo. I would stick to my decision to not include a fully worked out example of fine-tuning, freezing parameters, etc. It should be the subject of a separate tutorial where its usefulness is demonstrable and demonstrated.

It is the Gluon equivalent of this Module tutorial: https://github.com/apache/incubator-mxnet/blob/master/docs/tutorials/python/predict_image.md and this Module blog post: https://aws.amazon.com/fr/blogs/machine-learning/use-pre-trained-models-with-apache-mxnet/

safrooze · 2018-05-17T17:07:13Z

Agreed.

safrooze · 2018-05-17T17:10:11Z

docs/tutorials/gluon/pretrained_models.md

+
+# Using pre-trained models in MXNet
+
+In this tutorial we will see how to use multiple pre-trained models with Apache MXNet. First, let's download three images classification models from the Apache MXNet [Gluon model zoo](https://mxnet.incubator.apache.org/api/python/gluon/model_zoo.html).


Typo: images->image
Also since you're not really downloading any models in this section, maybe reword to: *In this tutorial we will use the following three different image classification models:"

safrooze · 2018-05-17T17:11:36Z

docs/tutorials/gluon/pretrained_models.md

+import mxnet as mx
+from mxnet import gluon, nd
+from mxnet.gluon.model_zoo import vision
+import matplotlib.pyplot as plt


PEP8: matplotlib < mxnet

safrooze · 2018-05-17T17:12:26Z

docs/tutorials/gluon/pretrained_models.md

+from mxnet.gluon.model_zoo import vision
+import matplotlib.pyplot as plt
+import numpy as np
+import json


PEP8: system imports (json) before other packages followed by a blank line.

safrooze · 2018-05-17T17:37:18Z

docs/tutorials/gluon/pretrained_models.md

+resnet18 = vision.resnet18_v1(pretrained=True, ctx=ctx)
+```
+
+We can look at the description of the MobileNet network for example, which has a relatively simple though deep architecture


"simple yet deep" or just simple deep?

safrooze · 2018-05-17T19:10:40Z

docs/tutorials/gluon/pretrained_models.md

+* Resize the shorter edge of the image 224.
+* Crop, using a size of 224x224 from the center of the image.
+* Shift the mean and standard deviation of our color channels to match the ones of the dataset the network has been trained on.
+* Reshape the array from (Height, Width, 3) to (3, Height, Width).


Transpose, not Reshape.

safrooze · 2018-05-17T19:12:04Z

docs/tutorials/gluon/pretrained_models.md

+                                      mean=mx.nd.array([0.485, 0.456, 0.406]),
+                                      std=mx.nd.array([0.229, 0.224, 0.225])) 
+    # the network expect batches of the form (N,3,224,224)
+    flipped_axis = normalized.transpose((2,0,1))  # Flipping from (224, 224, 3) to (3, 224, 224)


Maybe a better name than "flipped_axis"? I don't really think of transpose as flipping. You're not reversing that axis to consider it a flipped axis. Maybe swapped_axis? Also perhaps fix the comment as well to match.

safrooze · 2018-05-17T19:16:12Z

docs/tutorials/gluon/pretrained_models.md

+```
+
+## Testing the different networks
+We run the image through each pre-trained network. The models output a *NDArray* holding 1,000 activation values, which we convert to probabilities using the `softmax()` function, corresponding to the 1,000 categories it has been trained on. The *NDArray* has only one line since batch size is equal to 1


"The NDArray has only one line..." to "The output prediction NDArray has only one row..."

safrooze · 2018-05-17T19:25:40Z

docs/tutorials/gluon/pretrained_models.md

+
+As you can see, pre-trained networks produce slightly different predictions, and have different run-time. In this case, MobileNet is almost **5 times faster** than DenseNet!
+
+## Fine-tuning using pre-trained models


Fine-tuning pre-trained models (drop using)

safrooze · 2018-05-17T19:46:12Z

docs/tutorials/gluon/pretrained_models.md

+## Loading the model
+
+The [Gluon Model Zoo](https://mxnet.incubator.apache.org/api/python/gluon/model_zoo.html) provides a collection of off-the-shelf models. You can get the ImageNet pre-trained model by using `pretrained=True`. 
+If you want to train on your own classification problem from scratch, you can get an untrained network with a specific number of classes using the `classes` parameter: for example `classes=10`.


Maybe add a line of code to show how to get the untrained model with different number of classes? Also perhaps emphasise that one cannot change number of classes and user pretrained as the same time.

indhub · 2018-05-17T23:24:31Z

docs/tutorials/gluon/pretrained_models.md

+plt.imshow(image.asnumpy())
+```
+
+![png](https://github.com/dmlc/web-data/blob/master/mxnet/doc/tutorials/onnx/images/dog.jpg?raw=true)


notebook-skip-line here?

safrooze

LGTM

* adding the pre-trained model tutorial * adding pretrained model tutorial * updating the tutorial * Update pretrained_models.md * Update test_tutorials.py * Update pretrained_models.md * Update pretrained_models.md * updates following indhu's review * Update pretrained_models.md * Trigger build * Trigger build * Trigger build * implement sina feedback * Update pretrained_models.md * Trigger build

ThomasDelteil added 4 commits May 15, 2018 14:48

adding the pre-trained model tutorial

3595fb9

adding pretrained model tutorial

242f249

updating the tutorial

bd8d831

Update pretrained_models.md

225279b

ThomasDelteil requested a review from szha as a code owner May 15, 2018 22:06

Update test_tutorials.py

3be2761

ThomasDelteil commented May 15, 2018

View reviewed changes

ThomasDelteil added 2 commits May 15, 2018 15:17

Update pretrained_models.md

35faa6e

Update pretrained_models.md

119d372

indhub reviewed May 16, 2018

View reviewed changes

updates following indhu's review

8391ae7

ThomasDelteil added 2 commits May 16, 2018 21:25

Update pretrained_models.md

9b178fc

Trigger build

4ef388c

Trigger build

9580ab2

Trigger build

219c23f

indhub approved these changes May 17, 2018

View reviewed changes

safrooze reviewed May 17, 2018

View reviewed changes

implement sina feedback

84d0405

indhub reviewed May 17, 2018

View reviewed changes

Update pretrained_models.md

a16a395

safrooze approved these changes May 18, 2018

View reviewed changes

Trigger build

3d002d8

indhub merged commit 6abd654 into apache:master May 18, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[MXNET-423] Gluon Model Zoo Pre Trained Model tutorial #10959

[MXNET-423] Gluon Model Zoo Pre Trained Model tutorial #10959

ThomasDelteil commented May 15, 2018

piiswrong commented May 15, 2018

ThomasDelteil commented May 15, 2018 •

edited

ThomasDelteil May 15, 2018

indhub left a comment

indhub May 16, 2018

indhub May 16, 2018

indhub May 16, 2018

indhub May 16, 2018

ThomasDelteil commented May 16, 2018 •

edited

safrooze commented May 17, 2018

ThomasDelteil commented May 17, 2018 •

edited

safrooze commented May 17, 2018

safrooze May 17, 2018

safrooze May 17, 2018

safrooze May 17, 2018

safrooze May 17, 2018

safrooze May 17, 2018

safrooze May 17, 2018

safrooze May 17, 2018

safrooze May 17, 2018

safrooze May 17, 2018

indhub May 17, 2018 •

edited

safrooze left a comment

		* [Linear Regression](http://gluon.mxnet.io/chapter02_supervised-learning/linear-regression-gluon.html) <img src="https://upload.wikimedia.org/wikipedia/commons/6/6a/External_link_font_awesome.svg" alt="External link" height="15px" style="margin: 0px 0px 3px 3px;"/>
		* [MNIST Handwritten Digit Classification](/tutorials/gluon/mnist.html)



		Neural network expects input in a specific format. Usually images comes in the `Width x Height x Channels` format. Where channels are the RGB channels.
		This network accepts images in the `BatchSize x 3 x 224 x 224`. `224 x 224` is the image resolution, that?s how the model was trained. `3` is the number of channels : Red, Green and Blue (in this order). In this case we use a `BatchSize` of `1` since we are predicting one image at a time.


		# Using pre-trained models in MXNet

		In this tutorial we will see how to use multiple pre-trained models with Apache MXNet. First, let's download three images classification models from the Apache MXNet [Gluon model zoo](https://mxnet.incubator.apache.org/api/python/gluon/model_zoo.html).


		As you can see, pre-trained networks produce slightly different predictions, and have different run-time. In this case, MobileNet is almost 5 times faster than DenseNet!

		## Fine-tuning using pre-trained models

[MXNET-423] Gluon Model Zoo Pre Trained Model tutorial #10959

[MXNET-423] Gluon Model Zoo Pre Trained Model tutorial #10959

Conversation

ThomasDelteil commented May 15, 2018

Description

Checklist

Essentials

piiswrong commented May 15, 2018

ThomasDelteil commented May 15, 2018 • edited

Choose a reason for hiding this comment

indhub left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ThomasDelteil commented May 16, 2018 • edited

safrooze commented May 17, 2018

ThomasDelteil commented May 17, 2018 • edited

safrooze commented May 17, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

indhub May 17, 2018 • edited

Choose a reason for hiding this comment

safrooze left a comment

Choose a reason for hiding this comment

ThomasDelteil commented May 15, 2018 •

edited

ThomasDelteil commented May 16, 2018 •

edited

ThomasDelteil commented May 17, 2018 •

edited

indhub May 17, 2018 •

edited