Image Captioning Example

[shiffman]

There is interest in my A2Z class re: image captioning. I thought I would try to create a simple example modeled after the deeplearn.js imagenet that provides captions for a p5.Image as well as real-time captions of a recorded or live video. @cvalenzuela, just wanted to check whether I'd be duplicating anything you have already done? Will leave this thread open to discuss / plan.

[cvalenzuela]

Nice! I haven't started anything with images yet.
Maybe porting this https://github.com/tensorflow/models/tree/master/research/im2txt will be interesting too!

[cvalenzuela]

This is now implemented using SqueezeNet

Here is a simple example
And here is another one with a webcam

Should we use different models?

[shiffman]

Yes, definitely! It also might be nice like with the LSTM examples to offer a README on how to train your own model. . . although ultimately if students want to train models for image classification, some transfer learning process will likely be more workable in terms of creative coding scenarios.

[shiffman]

I've been working on the ImageNet/SqueezeNet examples. I've changed the following:

• predict() returns an array of classes sorted by probability
• predict() takes an optional 3rd argument for number of classes to return

The main thing I'd like to work through now is how to make this simpler to use with p5.Element and/or p5.Image. As of now example code requires createImg() and setting width and height attributes. @cvalenzuela does it only work with square, odd number resolutions? I think it would be nice for the library to internally handle any resizing of the image as well as accept a variety of image inputs direct from p5.

• predict() should accept p5.Image
• predict() should accept p5.Element that is an <img>.
• predict() should internally do any image resizing and let the user keep the image sized as it was for displaying.

Also, I think the setTimeout() if SqueezeNet isn't ready might be problematic. Probably best to return "not ready yet" message or something?

• rethink what predict() does when model not ready.

Also, the webcam example is broken, I will fix.

• fix webcam example.

@cvalenzuela feel free to fix up anything in 85c7311 if it's not conforming to the style you adopted.

Also, what's a better word than "label" for a class prediction? I was afraid to use "class" given it's a JS reserved word?

Thoughts?

[cvalenzuela]

• I haven't tried inputting images of different sizes. My guess is that should work too.
  Are you thinking of handling the resize/scale of images with p5 or plain js?

• Yes, the setTimeout() is an issue. 85c7311 looks good!

• Maybe category? That's what Google API calls it

Let me know if you need help with any of this

[shiffman]

• category is great. I can update that.
• I wasn't able to get it to work with different image resolutions but I can investigate more, I didn't try for long and might have made silly mistakes.
• re: p5 or plain js, I don't know! Ideally I think the predict function could take any of the following:
  • p5.Image
  • p5.Element (either video or image or canvas)
  • p5.Renderer (i.e result of createGraphics() )
  • native JS DOM element <img> or <video> or <canvas>
  • array of pixels?

[shiffman]

Adding:

• constructor should have 2nd optional callback for when model is ready. (If not using preload()).

[shiffman]

Noting here that passing in the optional third argument for number of categories is broken at the moment.

[cvalenzuela]

This seems resolved. #92 and #27 are following updates on this in separate threads.