Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP until the conversion issue is resolved] Add EfficientNet #238

Closed
wants to merge 37 commits into from

Conversation

@sdll
Copy link
Contributor

sdll commented Jun 16, 2019

This PR adds the implementation of EfficientNet B0-5 inference pipeline, with the support for quantized weights.
demo

Roadmap

  • Translate predicted logits to top K classes

  • Test for memory leaks and validate inference

  • Add the demo

  • Improve the docs

  • Make a script converting original checkpoints to TF.js JSON
    UPDATE 18/06: I have made a PR with the script to the most popular Keras implementation.
    UPDATE 25/06: The conversion tf.keras -> SavedModel -> TF.js Graph Model works well.
    UPDATE 08/07: The conversion to Keras in the first step turned out to be unnecessary, so I simplified the script, but could not handle a pesky bug with the converter (exporting to Keras first did not help as well, so the optimisation was not the source of the issue).

  • Integrate the TF Checkpoint to Keras converter script into the workflow for end-to-end porting of weights to TF.js JSON
    UPDATE 18/06: To make the model work, we need to implement two custom layers and an initializer.

  • Fix the conversion issues

cc: @manrajgrover


This change is Reviewable

@googlebot googlebot added the cla: yes label Jun 16, 2019
sdll added 2 commits Jun 16, 2019
sdll
sdll
@sdll sdll mentioned this pull request Jun 22, 2019
2 of 20 tasks complete
Sasha Illarionov added 9 commits Jun 25, 2019
Sasha Illarionov
* upstream/master:
  Update posenet lock file. (#244)
  PoseNet - Split pose estimation methods  into estimateSinglePose and estimateMultiplePoses, and move inputResolution specification from loading to inference time. (#242)
  update peer deps for toxicity model and increase the minor version (#243)
  update models peerDependencies to be individual npm instead of union (#237)
  [speech-commands] Bump version to 0.3.9 (#240)
  [speech-commands] Enable creating recognizer from artifacts (#226)
  Update README.md (#239)
  export Mobilenet as interface (#236)
  skip unneeded operation for posenet single-pose
  allow user to provide a custom url for the mobilenet model (#235)
Sasha Illarionov
Sasha Illarionov
Sasha Illarionov
Sasha Illarionov
@sdll sdll changed the title [WIP][**do not review**] Add EfficientNet [WIP] Add EfficientNet Jun 28, 2019
@sdll sdll force-pushed the sdll:models/EfficientNet branch from bdd3f92 to 94aea5c Jun 29, 2019
Sasha Illarionov added 8 commits Jun 29, 2019
Sasha Illarionov
Sasha Illarionov
* upstream/master:
  Optimize the bundle of our demos (#247)
  Fix the broken gif in PoseNet. (#246)
Sasha Illarionov
Sasha Illarionov
Sasha Illarionov
@sdll sdll changed the title [WIP] Add EfficientNet Add EfficientNet Jun 30, 2019
Sasha Illarionov added 2 commits Jul 2, 2019
Sasha Illarionov
Sasha Illarionov
Copy link

manrajgrover left a comment

Hi @sdll,

Awesome work here! 💯 I've partially reviewed the PR. There seems to be an issue in conversion script.

Kindly look into the same and I'll review the rest of the PR once fixed.

efficientnet/README.md Outdated Show resolved Hide resolved
efficientnet/README.md Outdated Show resolved Hide resolved
efficientnet/README.md Outdated Show resolved Hide resolved
efficientnet/README.md Outdated Show resolved Hide resolved
efficientnet/README.md Show resolved Hide resolved
efficientnet/README.md Outdated Show resolved Hide resolved
CONVERTED_MODELS_DIR="efficientnet"
VIRTUALENV_DIR="venv"
MODEL_SOURCE_DIR="efficientnet-source"
MODEL_SOURCE_URL="https://github.com/sdll/efficientnet"

This comment has been minimized.

Copy link
@manrajgrover

manrajgrover Jul 7, 2019

We should source the checkpoint from an official tfjs bucket and remove dependence on the forked repository

},
MEAN_RGB: [0.485, 0.456, 0.406],
STDDEV_RGB: [0.229, 0.224, 0.225],
IMAGENET_CLASSES: {

This comment has been minimized.

Copy link
@manrajgrover

manrajgrover Jul 7, 2019

I think we should move these classes to a separate JSON file and import it here.

This comment has been minimized.

Copy link
@sdll

sdll Jul 7, 2019

Author Contributor

This seems to be more natural, since otherwise we have to cast class labels to integers, losing the information stored in the data itself.

efficientnet/demo/src/index.js Outdated Show resolved Hide resolved
@@ -0,0 +1,216 @@
#!/usr/bin/env bash

This comment has been minimized.

Copy link
@manrajgrover

manrajgrover Jul 7, 2019

This script throws the following error

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~ ............................. ~
~ Converting efficientnet-b0... ~
~ ............................. ~
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Traceback (most recent call last):
  File "/home/manrajsingh/github/tfjs-models/efficientnet/dist/efficientnet-source/scripts/load_efficientnet.py", line 156, in <module>
    import eval_ckpt_main
ModuleNotFoundError: No module named 'eval_ckpt_main'

Kindly look into the same

This comment has been minimized.

Copy link
@sdll

sdll Jul 7, 2019

Author Contributor

Recent updates to tensorflowjs_converter have introduced a bug which I could not fix for hours. I have tried to roll back (and forth!), but to no avail. The issue is explained here.

Sasha Illarionov added 4 commits Jul 7, 2019
Sasha Illarionov
…o models/EfficientNet
Sasha Illarionov
@tafsiri

This comment has been minimized.

Copy link
Member

tafsiri commented Jul 11, 2019

@sdll A few quick questions. Apart from the conversion issues (I'll try to reproduce that locally when I get a chance), I did have some issues running the build (incorrect reference to preamble) and running the demos (no keys in efficientnets variable thus non of the buttons work). I did hack it in, but didn't get a meaningful prediction. Are those parts still work in progress?

At a higher level I was wondering how big the weights are (in MB)? And what kind of inference times you are getting?

@sdll sdll changed the title Add EfficientNet [WIP] Add EfficientNet Jul 12, 2019
@sdll

This comment has been minimized.

Copy link
Contributor Author

sdll commented Jul 12, 2019

Sorry, @tafsiri, for the confusion, I glossed over changes I had to do following those based on @manrajgrover’s suggestions. This is fixed now, I have successfully published the module locally and tested each of the variants. Here is the screenshot:

Screen Shot 2019-07-12 at 5 09 18 PM

Let me know if there are any issues other than those outlined here.

Models quantized to 2 bytes weigh from 10.2 MB (b0) to 58.2 MB (b5).

With regards to performance, b0 is 4.8 s on the cold first run and 0.52 s on the second. The b5 variant is slower, taking 7.9 s on the first run, and 3.2 s on the second.

I will refactor the code to follow the factory pattern of loading the model, taking inspiration from your review of DeepLab.

Thanks for your time and help!

Copy link
Contributor Author

sdll left a comment

Reviewable status: 0 of 1 approvals obtained (waiting on @manrajgrover and @sdll)


efficientnet/README.md, line 21 at r1 (raw file):

Previously, manrajgrover (Manraj Singh) wrote…
const efficientnet = new EfficientNet(model, isQuantized);

Done.


efficientnet/README.md, line 28 at r1 (raw file):

Previously, manrajgrover (Manraj Singh) wrote…

This should be a relative URL to the source file.

Done.


efficientnet/README.md, line 48 at r1 (raw file):

Previously, manrajgrover (Manraj Singh) wrote…

We should also pass topK here as an example.

Done.


efficientnet/README.md, line 58 at r1 (raw file):

Previously, manrajgrover (Manraj Singh) wrote…
EfficientNets rely on AutoML and compound scaling to achieve superior performance without compromising resource efficiency. The [AutoML Mobile framework](https://ai.googleblog.com/2018/08/mnasnet-towards-automating-design-of.html) has helped develop a mobile-size baseline network, **EfficientNet-B0**, which is then improved by the compound scaling method to obtain EfficientNet-B1 to B7.

Done.


efficientnet/README.md, line 60 at r1 (raw file):

Previously, manrajgrover (Manraj Singh) wrote…

Let's format this html code

Done.


efficientnet/README.md, line 82 at r1 (raw file):

Previously, manrajgrover (Manraj Singh) wrote…
./scripts/convert_efficientnet.sh --target_dir=dist --use_venv=t

We will need to fix this after the issue with the converter is resolved (might need to switch to the docker-based pipeline to avoid issues on OS X, for example).


efficientnet/README.md, line 88 at r1 (raw file):

Previously, manrajgrover (Manraj Singh) wrote…
./scripts/convert_efficientnet.sh -h

Done.


efficientnet/demo/src/index.js, line 27 at r1 (raw file):

Previously, manrajgrover (Manraj Singh) wrote…

Let's add keys to Object dynamically during initialization.

Done.


efficientnet/scripts/convert_efficientnet.sh, line 145 at r1 (raw file):

Previously, manrajgrover (Manraj Singh) wrote…

We should source the checkpoint from an official tfjs bucket and remove dependence on the forked repository

Done.

@sdll sdll force-pushed the sdll:models/EfficientNet branch from 1d1733c to 79e37f5 Jul 12, 2019
@sdll sdll changed the title [WIP] Add EfficientNet [WIP until the conversion issue is resolved] Add EfficientNet Jul 12, 2019
Copy link
Contributor Author

sdll left a comment

Reviewable status: 0 of 1 approvals obtained (waiting on @manrajgrover and @sdll)


efficientnet/src/types.ts, line 25 at r5 (raw file):

    |ImageData|HTMLImageElement|HTMLCanvasElement|HTMLVideoElement|tf.Tensor3D;

// #TODO: Fix this after the issues with the converter are resolved

After the conversion issue is resolved, I will add support for all kinds of quantization.

@tafsiri

This comment has been minimized.

Copy link
Member

tafsiri commented Jul 15, 2019

@sdll just fyi i get the same issue with calls to hasLoaded() throwing an error that i do in the deeplab demo.

Thanks for the model info

@sdll

This comment has been minimized.

Copy link
Contributor Author

sdll commented Jul 16, 2019

@tafsiri, a fresh install did not show any issues:

# cd tfjs-models/efficientnet
# git pull
cd demo
rm -rf node_modules/ .yalc/ .cache/
cd ..
yarn publish-local
cd demo
yarn
yarn link-local
yarn watch

One thing that I did notice though was that labels were off in Chrome due to the differences in CSS rendering:

Screen Shot 2019-07-16 at 12 27 05 PM

This is fixed now:

Screen Shot 2019-07-16 at 12 45 33 PM

@sdll sdll closed this Jul 29, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

4 participants
You can’t perform that action at this time.