Generic Neural Network class

[shiffman]

Following up on #483

→ Description 📝

Submitting a new feature 🆕

I am going to start working on the generic neural network class in this branch in case anyone wants to follow!

→ Relevant Example or Paired Pull Request to ml5-examples 🦄

ml5js/ml5-examples#164

→ Relevant documentation 🌴

Documentation to come!

[jonnytest1]

just gonna leave this here since its fairly similar and might help out
https://github.com/jonnytest1/tpscripts/blob/master/learning/tensorflow.js
(ignore the stuff at the top thats just some custom injection i wrote)

[shiffman]

I've implemented a "proof of concept" for training and prediction with this generic Neural Network class in 37414d7. Here is some sample code that I am using for a manual test. I'm going to add a separate comment for TODOs and questions momentarily.

// Create Neural Network
const neuralNetwork = ml5.neuralNetwork(3, 2);

// Add training data
const training_input = [-0.6, 1, 0.25];
const training_target = [0.3, 0.9];
for (let i = 0; i < 500; i++) {
  neuralNetwork.addData(training_input, training_target);
}

// Train
neuralNetwork.train(20, whileTraining);

// Training callback
function whileTraining(error, progress) {
  if (progress.status == 'training') {
    console.log(progress.epoch, progress.loss);
  } else if (progress.status == 'complete') {
    // Run prediction when complete
    let input = [-0.6, 1, 0.25];
    neuralNetwork.predict(input, gotResults);
  }
}

function gotResults(error, results) {
  console.log(results.output);
}

[shiffman]

TODO:

• implement callbacks + promises properly with ml5's callCallback().
  • train function uses ml5's callCallback().
  • predict function uses ml5's callCallback().
• check for shape when data is sent in for training and prediction.
• add option to make a "classifier" (change to softmax activation and categoricalCrossEntropy)
• finalize defaults / heuristics for hidden layer and activation functions and more. . .
• memory management
• load and save functions
• implement neuro-evolution functions

QUESTIONS:

• Do we need a separate ml5.data a la tf.data?
• Do we allow for multiple hidden layers and if so, how?
• To what extent for these very simple models should we be wrapping tf.layers vs. just doing the matrix math ourselves (this is perhaps more relevant for my intended use of this with neuro-evolution examples and possible the reinforcement learning work of @AidanNelson.)

@jonnytest1 thank you for your comments and feedback! My goal for this implementation is to create something simpler and more basic from the the usual technique (like in keras / tf.layers) of manually configuring and adding the layers one at a time. The options object would allow for some customization, but ultimately this will serve as a stepping stone to using tf.layers directly for custom architectures!

[shiffman]

TODO: add activation function to options

[yining1023]

added

[shiffman]

TODO: what's a good heuristic for hidden units?

[shiffman]

TODO: add to options, pick a default

[yining1023]

Hard to decide a default LEARNING_RATE for generic neural nets, tensorflow playground defaults to 0.03. Should we pick 0.1?

[shiffman]

TODO: what's a good default? Should be categoricalCrossEntropy if we add a "classification" type to options?

const options = {
  task: 'classification'
}

[shiffman]

needs to use ml5 callCallback()

[yining1023]

Added ml5 callCallback().

[shiffman]

This function is a little wonky b/c it calls the callback continuously. . so that is required here the way I've implemented it. We could use callCallback() to support a promise for when it is finished training?

[yining1023]

Maybe we could have two kinds of callbacks: whileTraining and doneTraining? As I tried to implement supporting promise and callback, it's hard to keep one callback getting called all the time, and use a promise to return results when it's done training. Then I realize that it's clearer to have separate callbacks. They are both optional.

// Use both callbacks
neuralNetwork.train(20, whileTraining, doneTraining);

// Use promise
const trainingResults = await neuralNetwork.train(20);
// trainingResults = { status: 'complete' }

// Use whileTraining callback
const trainingResults = await neuralNetwork.train(20, whileTraining);

// User doneTraining callback
neuralNetwork.train(20, null, doneTraining);

[shiffman]

Not sure what the name of this function should be or whether it's ml5's job really to compile the data. However, this I view as analogous to addExample() with KNN or featureExtractor transfer learning. It's possible that we'll want to have a ml5.data feature for the library. At the moment I am considering "batches" to be beyond the scope of ml5 and handled internally.

[shiffman]

Add a check here to see if the user is sending one sample or multiple samples. Wrap in an array if one sample, otherwise it's already the right shape.

[joeyklee]

@shiffman - Making a note about loading models which might create some problems at the moment.

In the case that a pre-trained model is loaded, running .predict() or .classify() will fail because we don't have the inputMin, inputMax, outputMin, outputMax values that are looked up when those functions are called.

As a result, given the current structure of the NeuralNetwork, loading pretrained models won't work as we hope unless users specify those values in the normalizationOptions.

I have to think about how to handle this. Any thoughts here are welcome!

[shiffman]

Hi @joeyklee, aha, yes, this makes sense!

What about including these values as part of the metadata in model.json? I believe there is some precedent for this in the way that we have added extra metadata to transfer learning image models. I would prefer not to introduce an additional metadata.json file, but this is also an option. I recall there has been some hesitation to mess with model.json in the past as it could (?) cause problems with generic tf.js compatibility. I would say this isn't a big issue since these models wouldn't work well outside of ml5 without the data helper functionality anyway.

[joeyklee]

@shiffman - yep! Just made a note here: ml5js/ml5-examples#210 (comment)

[joeyklee]

@shiffman - Ok! The model loading issue should be addressed now in: #613

[shiffman]

Noting that load('model.json') doesn't work in the web editor, I think the meta file is not getting the path passed to it?

https://editor.p5js.org/ima_ml/sketches/Etmls1Eq-

[joeyklee]

@shiffman - One thing is that in the web editor it looks like the /model sub director is dropped from the URL.

So it looks like the way the p5 editor handles the reference to the model file looks like this:
The file path that is given:
blob:https://editor.p5js.org/d7bdc1c9-5d3e-ad4d-950d-6941a0410be8
the model meta json is being searched for here:
blob:https://editor.p5js.org/model_meta.json

So when the reference to the model/model.json is sent in, what we get is this URL to the mongo id of that file. Where as, our fetch request that is occurring within ml5 gets all wonky because the file path being sent in is actually a reference to something else and not the actual path.

Update:

Just had a chat with @catarak and it sounds like we have some options:

1. The model.json can live at an absolute path somewhere else with the model_meta.json and the model.weights.bin
2. we can add in a second and third parameter to the .load() function or an array of paths that point those files of interest

I think I'd go for options number 2 to support working in the web editor. I'll make a PR for this to propose some changes.

[shiffman]

Everybody turn your keys and get ready for launch, in 3 .. 2 ... 1 ...

🚀🚀🚀
🌈🌈🌈
🦄🦄🦄