Different inference results

[forete]

Hi, I was testing Teachable machine on the browser and good pretty good results for a personal project. However, when downloading the generated model to Keras and using the code snippet that you provide, the inference results are totally different with the same test set. Could it be because the Keras snippet is resizing the images in a different way to the teachable machine application in the browser? Any ideas on how to fix it?
Many thanks for your help!

[plutorain]

I had same issue
I made two classes and It can works well on website.
But when I download tensorflow.js model, the model works well.
But keras model can't distinguish class
(in my case, The keras model shows high probability only for one class)

[HalfdanJ]

Any chance you can share an example where this is clear? Both the teachable machine project and the python code you use to validate?

We have had a suspicion that there can be some difference, but we dont have a test dataset that clearly indicates it.

[GautamBose]

Hi! We have recently updated the snippet available in Teachable Machine's Export panel to resize the images exactly how they are resized in the web app. Check it out and see if your classification performance improves.

[AliKarpuzoglu]

I reproduced the error @HalfdanJ and @GautamBose.
I tried it with both your version and with the fixed normalization I have in Pullrequest #81
Class 2-samples (2).zip
Class 1-samples (1).zip

Teachable machine returns 49-51 Class-1 Class-2
And Keras returns [0.546911 0.45308906] - or [0.5456604 0.45433962] with the new normalization
Should I upload the model, too?

[GautamBose]

Hi! I suspect that something is causing models trained on different OS's, hardware, or browsers to behave differently. For example, when I downloaded your dataset and trained my own model, Teachable Machine gave me 78-22 Class-1 Class-2 on that test image... and keras returns [0.7885991 0.21140094], with the original normalization.

This is on MacOS 10.14.16 and Chrome 80.0.3987.116

To get to the bottom of this, what OS / browser are you using?

[AliKarpuzoglu]

Mac OS 10.15.2 (19C57) and Chrome 80.0.3987.106 (64-bit)

[AliKarpuzoglu]

So I have tried training it again, and I got a result of 60-40
So the second time I got a result of 89 -11
The third time I got a 33 - 67
It makes sense that we have some randomness in the way you guys probably shuffle the image before training. ( I didn't look into the source code )

Edit: Now I did look into the source code, you shuffle it here, which causes the obvious deviations when running the same thing multiple times, It is weird that the second and third try had much smaller deviations though.

teachablemachine-community/libraries/image/src/teachable-mobilenet.ts

Line 209 in 9c0c104

private convertToTfDataset() {

I used Colab for Keras, did you run it locally?

So I ran all of them in Colab and got this :

1. 60- 40 -> [[0.831798 0.16820198]]
2. 89- 11 -> [[0.906152 0.09384797]] (this ones good)
3. 33 - 67 -> [[0.3598008 0.6401992]]

Colab uses : Python 3.6.9 and Ubuntu 18.04.3 LTS

[AhmetEnesYalcinkaya]

Is there any solution for this problem ? How can I get better result ?

[TowhidKashem]

Having the same issue here, UI results are very accurate and in the form of x: 10%, y: 90% but the downloaded files using the provided code snippet for tensorflow.js are very different in the form of x: 0.47, y: 0.53 for the same test images