add image classifier model

[OrpheasK]

Bringing in the ImageClassifier model, this branch picks up from where @gohai made the necessary changes regarding input normalization. This version alters the way video classification is used, introducing classifyStart() for a similar approach to image input, instead of passing video input as an argument to the constructor. Many thanks to @ziyuan-linn for filling me in on alterations that had taken place while migrating to the next gen version of the library.

[gohai]

Hi @OrpheasK - kudos to getting this to a mergable state! I'll quickly look through the changed files and add comments if something jumps out. Best from Shanghai

[gohai]

It might be more beginner friendly to do this without template literals here.

[gohai]

This file looks like it might have been accidentally included?

[gohai]

We explored "overrides" before @ziyuan-linn made us switch to yarn, where this doesn't seem to be necessary anymore. Probably good to delete this, especially if it installs on yarn without it.

[shiffman]

This is looking fantastic @OrpheasK, yay! Once this is merged I think we'll be able to make this repo the official new ml5.js release! I quickly tested the video example with a teachable machine model, but got an error. Have you tried TM models yet? Here's a URL for testing:

function preload() {
  classifier = ml5.imageClassifier(
    "https://teachablemachine.withgoogle.com/models/bXy2kDNi/model.json"
  );
}

[OrpheasK]

Thank you for the notes @gohai and @shiffman, I will look into these right away! The bulk of testing I did has been with Mobilenet at this point, I will make sure TM also runs smoothly to touch upon the main user scenarios of the Image Classifier.

[OrpheasK]

This commit enables the Image Classifier functionality with Teachable Machine models. It also introduces classifyStop() to be used along with classifyStart() for video classification. On that end, the flags isClassifying, signalStop, prevStart are added to maintain a consistent structure of code with the other ml5 models which make use of detectStart()/detectStop().

All models included are functional with Mobilenet and Teachable Machine tested more thoroughly. Some of them still require image resizing and for those imgToTensor() is applied to the input similarly to the previous version of ml5.

[lindapaiste]

It is so frustrating to see all the work that I have put into this library being completely ignored. I rewrote this utility to be cleaner, more efficient, better documented, and handle more situations. Why are we still using the old version?

https://github.com/ml5js/ml5-library/blob/a0ba76f5f11a1f39e5311985b55a4edb4f14d22b/src/utils/gettopkclasses.js

ml5js/ml5-library#1364

[lindapaiste]

This entire block is not necessary because img is already a Tensor. I explained that and I fixed it in ml5js/ml5-library#1396 but no one is even looking!?!

[shiffman]

Thank you @lindapaiste for highlighting this! I'll be in touch to coordinate a meeting where we can look at the best way to integrate the work you did in the previous repo. Thank you for your patience with us!

[shiffman]

@OrpheasK I left some nitpicky comments mostly about the examples. I tested and the examples are working well, thank you for this work! My apologies again to @lindapaiste, I'll be coordinating a meeting to discuss if it makes the most sense to merge this and then integrate Linda's work or whether we want to integrate it into this branch before merging. Thanks everyone!

[shiffman]

nitpicking comment, but i would put the modifiers for drawing the text in draw()

[shiffman]

Using video is perhaps more clear?

[shiffman]

Maybe the basic example doesn't need to show how to stop the classifier, just with a mouse press might be confusing to the user.

[shiffman]

I would take out the <h1> elements as they can be confusing for beginners especially if the primary environment is the p5.js web editor and they aren't used to editing index.html.

[shiffman]

Thank you @lindapaiste for highlighting this! I'll be in touch to coordinate a meeting where we can look at the best way to integrate the work you did in the previous repo. Thank you for your patience with us!

[lindapaiste]

@OrpheasK here's a plain JS version of the pre-processing utilities from ml5js/ml5-library#1398 the rest of that PR is already in JS and should be usable.

import * as tf from '@tensorflow/tfjs';

/**
 * Common operations for pre-processing and post-processing images.
 */

/**
 * Rescale the pixel values from one range to another.
 *
 * @param {tf.Tensor} tensor
 * @param {number} inMin
 * @param {number} inMax
 * @param {number} outMin
 * @param {number} outMax
 * @return {tf.Tensor}
 */
function rescale(tensor, inMin, inMax, outMin, outMax) {
  // Note: could be one chained operation if useless ops like .sub(0) .div(1) are not a concern.
  return tf.tidy(() => {
    let result = tensor.toFloat();
    if (inMin !== 0) {
      result = tf.sub(result, tf.scalar(inMin));
    }
    const divisor = (inMax - inMin) / (outMax - outMin);
    if (divisor !== 1) {
      result = tf.div(result, tf.scalar(divisor));
    }
    if (outMin !== 0) {
      result = tf.add(result, tf.scalar(outMin));
    }
    return result;
  });
}

export default class ImageProcessor {
  /**
   * @typedef {Object} ProcessorConfig
   * @property {[number, number, number] | [number, number, number, number]} inputShape
   * @property {[number, number]} inputRange - Value range expected by the model. Typically [-1, 1] or [0, 1].
   * @property {boolean} [alignCorners] - default `false`
   * @property {'bilinear' | 'nearest'} [resizeMethod] - default 'bilinear'
   *
   * @param {ProcessorConfig} config
   */
  constructor(config) {
    /**
     * @type {ProcessorConfig} - The options which were passed to the constructor.
     * @readonly
     */
    this.config = config;
    // Derive size and numChannels from the inputShape.
    const shape = config.inputShape;
    const [height, width, numChannels] = shape.length === 3 ? shape : shape.slice(1);
    /**
     * @type {number} - The depth of the expected image. Can be 1 (grayscale), 3 (RGB) or 4 (RGBA).
     * @readonly
     */
    this.numChannels = numChannels;
    /**
     * @type {[number, number]} - The image input size expected by the underlying model.
     * @readonly
     */
    this.size = [height, width];
  }

  /**
   * @param {tf.Tensor3D|tf.Tensor4D} input
   * @return {tf.Tensor3D|tf.Tensor4D}
   */
  resize(input) {
    // Skip no-op resize.
    if (input.shape[0] === this.size[0] && input.shape[1] === this.size[1]) return input;
    // Use appropriate method based on config.
    const resizeFn = this.config.resizeMethod === 'nearest' ? tf.image.resizeNearestNeighbor : tf.image.resizeBilinear;
    return resizeFn(input, this.size, this.config.alignCorners);
  }

  /**
   * Pre-process an image based on the provided config. Handles:
   *  - conversion to tensor
   *  - correct number of channels (alpha, grayscale)
   *  - resizing
   *  - scaling the range of values
   *  - converting to 4D batch (if needed)
   *
   * @param {tf.Tensor3D | HTMLVideoElement | HTMLCanvasElement | HTMLImageElement} input
   * @return {tf.Tensor3D|tf.Tensor4D}
   * @public
   */
  preProcess(input) {
    return tf.tidy(() => {
      // Convert to tensor, handle depth.
      const tensor = (input instanceof tf.Tensor) ? input : tf.browser.fromPixels(input, this.numChannels);
      // Resize width/height.
      const resized = this.resize(tensor);
      // Rescale values from range [0-255] to inputRange.
      const [min, max] = this.inputRange;
      const rescaled = rescale(resized, 0, 255, min, max);
      // Possible batch.
      return this.inputShape.length === 3 ? rescaled : rescaled.expandDims(0);
    });
  }

  // TODO: post-process, for models which return an image

}

[OrpheasK]

Thanks for this @lindapaiste, will be looking into the integration of #1398 along with the #1396 and #1362 PRs which were mentioned in the discussion!

[shiffman]

I've taken another look through this pull request and as we discussed, I think it makes sense to merged first and we can look at bringing in the enhancements, optimizations, and code cleanup proposed by @lindapaiste in separate branches, thank you for everyone's patience and understanding!

[lindapaiste]

This will call the this.classifyInternal function twice. The callCallback function returns the underlying promise so you can await it. You don't need line 209. You can just do await callCallback(this.classifyInternal(image, number), callback);

[OrpheasK]

Noted, thanks for catching this, will be corrected in the upcoming Image Classifier update

[lindapaiste]

In the detector models we are using await tf.nextFrame() and structuring the loop a bit differently. I don't know that one is necessarily better than the other but IMO we should be consistent.

[OrpheasK]

Will be looking into this