[GSoC 2019] Growing the model garden 🌳

[sdll]

This issue tracked the progress for GSoC'19 project "Reasonable Effectiveness of Mobile Inference: Adaptive Growth of the TensorFlow.js Model Garden".

The final report is here.

Active

• DeepLab

  • Status: Merged into the model garden.
  • Requires compute: no
  • PR
  • Reference implementation

• EfficientNet

  • Status: The port was completed, but the model did not provide 10x improvements over MobileNet and will be moved to the personal repo.
  • Requires compute: no
  • PR
  • Reference implementation

• Text Detection

  • Status: The js code is under review. The optimized model requires more training to pass the quality tests for the merge.
  • Requires compute: yes
  • Reference implementations: (tf.slim) (PyTorch)
  • PR

• Text Recognition

  • Reference implementation
  • Status: The port from pytorch to tf.keras is under development.
  • Requires compute: yes

Scratchpad

• VGGish

  • Status: Not in active development
  • Requires compute: no
  • Reference implementation
  • Usage Example

• DeepGaze

  • Status: Not in active development
  • Requires compute: yes
  • Reference implementation
  • Motivation: lookie-lookie

• Style Transfer

  • Status: Waiting for the core team feedback
  • Requires compute: no
  • Existing implementations: (1) (description & lines of improvement)

• Pix2Pix

  • Status: Waiting for the core team feedback
  • Requires compute: no
  • Reference implementation
  • Existing TF.js ports: (1)

• Speech to Text

  • Status: Not in active development
  • Requires compute: no
  • Reference implementation
  • Motivation

• Face Swap

  • Status: Not in active development
  • Requires compute: possibly
  • Reference implementation
  • Existing TF.js ports: (1)

• Sketch Simplification

  • Status: Not in active development, awaiting the decision on licensing
  • Requires compute: no
  • Reference implementation

• DeOldify

  • Status: Not in active development
  • Requires compute: yes
  • Reference implementation

• HNAtt

  • Status:
    • Waiting for the core team feedback
  • Requires compute: yes
  • Reference implementation

• VAMPIRE

  • Status: Not in active development
  • Requires compute: yes
  • Reference implementation
  • Motivation: Might be a good addition either to tfjs-models or tfjs-examples as an alternative to USE for domain-specific text analysis.

• CycleGAN

  • Status: Not in active development
  • Requires compute: yes
  • Reference implementation

• EDANet

  • Status: Not in active development
  • Requires compute: no
  • Reference implementation

• ShuffleNet

  • Status: Not in active development
  • Requires compute: yes
  • Reference implementation

• ErfNet

  • Status: Not in active development
  • Requires compute: yes
  • Reference implementation

• YOLO

  • Status: Not in active development
  • Existing TF.js port: (1)
  • Requires compute: no
  • Reference implementation

• Mask-RCNN

  • Status: Not in active development
  • Requires compute: yes
  • Reference implementation

[sdll]

@nsthorat (as I have promised here), @dsmilkov, @manrajgrover, let me know if you have any opinion on which models you would like to see first. I would love to hear any ideas and high-level views on the plan and details of implementation as well.

[kavikode]

Mask-RCNN and YOLO will be very helpful to see first @sdll @nsthorat @dsmilkov @manrajgrover because they are very useful and commonly used for computer vision. For my own research, I'm trying to use it for robotics control to provide accessibility people with disabilities. Among all of the other models, these models will increase accessibility control and independence for people with disabilities. In fact, I could really use some help to run Mask-RCNN and YOLO in the browser for robotics control as soon as possible. Hope you can please help.

[sdll]

@kavikode, TF.js implementations of YOLO already exist. To convert Mask-RCNN as described here, you might try the following:

• switch the source to tf.keras
• load the pre-trained weights
• export the model to tf.SavedModel
• convert tf.SavedModel using tfjs-converter

However, Mask-RCNN is quite a heavyweight model and may require further optimizations to run smoothly.

[kavikode]

thank you so much @sdll

[beriberikix]

What is the process to pitch for moving items from the scratchpad to the active list? I reported #1725 and love to make a case to whoever will listen :)

[sdll]

@beriberikix, thank you for your interest! I have not dived into the details, but DeepSpeech seems to be resource-intensive, given the discussion in your other issue. Have you had any success converting the model using tfjs-converter, how does it fare?

[beriberikix]

I haven't had any success with the converter tool. I've asked for some clarification on how the saved model is generated. I'll updated if and when I get the converter working.

[jessetrana]

I've worked some with DeepSpeech in the past. As I recall, it has both significant compute work for generating the input audio features (a la MFCC) as well as decoding via the language model. I'm not sure how much of that functionality also has corresponding JS libraries, so it's possible that porting it may be an involved effort. By any chance, have you looked into the supporting work apart from the TF model yet to see how easy or hard it might be?

[sdll]

Thanks everyone for the suggestions. Since the GSoC 2019 run came to the end, I have added the final report to the issue and updated the status of each port. Let me know if you have any other ideas on porting text detection/text recognition or would like to work on other models.