ResNet producing non deterministic and wrong predictions

[jeremiedb]

Predictions from the ResNet pre-trained models seems to have an issue. The predicted probabilities are changing at each call and essentially look like random. VGG19 does work fine however.

using Flux
using Metalhead

img = Metalhead.load("data/cats/cats_00001.jpg");

julia> classify(VGG19(), img)
"tiger cat"

julia> classify(VGG19(), img)
"tiger cat"

julia> classify(ResNet(), img)
"reel"

julia> classify(ResNet(), img)
"abacus"

julia> classify(ResNet(), img)
"spotlight, spot"

julia> classify(ResNet(), img)
"fountain"

I first though it might be an issue with a different preprocessing required for ResNet, thought from the randomness in the output of the model, I'd guess some layers didn't get properly defined, possibly the BatchNorm?

If using testmode!, resnet will then produces the same predictions after each run. However, a different set of weights seem to be initialized right at each call to ``

resnet = ResNet().layers
testmode!(resnet)
cat_pred_resnet = resnet(preprocess(RGB.(img)))
julia> findmax(cat_pred_resnet)
(1.0f0, CartesianIndex(900, 1))

cat_pred_resnet = resnet(preprocess(RGB.(img)))
julia> findmax(cat_pred_resnet)
(1.0f0, CartesianIndex(900, 1))

resnet = ResNet().layers
testmode!(resnet)
cat_pred_resnet = resnet(preprocess(RGB.(img)))
julia> findmax(cat_pred_resnet)
(0.99877006f0, CartesianIndex(413, 1))

[jeremiedb]

[ToucheSir]

I have a suspicion this and related issues are due to not all weights being properly loaded. See https://julialang.zulipchat.com/#narrow/stream/237432-ml-ecosystem-coordination/topic/ResNet.20weights for details, but TL;DR no batchnorm or bias! If it hasn't done so already, this should be fixed in #70 eventually.

[maxfreu]

Has the PR brought any change?

[darsnack]

Not other than discovering that the previous pre-trained model was wrong. I'm working on re-training the current models.

[maxfreu]

So re-training is all whats needed to fix it (hopefully)? Or is there still more to it? Outputs changing all the time sounds like BN layers are continuously adapting.

[ToucheSir]

Outputs shouldn't change constantly if the normalization layers are frozen, so I assume they weren't in the example above. This definitely shouldn't be a problem with the new models, because Flux.testmode! should work OOTB.

[darsnack]

Yeah this issue describes the old implementations. The new ones on master should not suffer from this issue. Training should be all that's necessary for a working model.

[maxfreu]

Do you need help with training more models? Where’s the bottleneck?

[darsnack]

Primarily the lack of a multi-GPU flow for DDP. Hopefully once the JuliaCon code is released, I can use that. For now, I'll just train with a single GPU.

[maxfreu]

If you provide the training fixture code, I can provide hardware and electrons :)

Edit: Sounds like I have to watch the juliacon videos!!

[trahflow]

so is this still relevant, or were the models re-trained in the meantime? (and if they were, did this fix the issue?)

[darsnack]

No models have been re-trained so far. I can help guide someone to set it up if they have a GPU. Pre-trained weights would be a welcome contribution.