About ResNet9 benchmark

[a-r-n-o-l-d]

Hi DrChainsaw (response to your question), for now I haven't replicate benchmark, I only use my ResNet9 implementation with my own dataset, not CIFAR10.
However, I will happily exchange my future results with you. I am currently stuck with last version Flux (v0.11.1), because when I try to use a convolutionnal layer without biases there is some GPU error.

I notice that your image normalization seems different from original benchmark (as far I understand your code properly). Maybe you could try something like that :

using MLDatasets
using Images
using Augmentor
using Statistics
using Flux: onehotbatch

# Compute train dataset statistics
if !isdefined(Main, :meanC10) || !isdefined(Main, :stdC10)
    r = CIFAR10.traintensor()[:,:,1,:]
    g = CIFAR10.traintensor()[:,:,2,:]
    b = CIFAR10.traintensor()[:,:,3,:]
    const meanC10 = [mean(r), mean(g), mean(b)]
    const stdC10 = [std(r), std(g), std(b)]
end

function normalize(img)
    tmp = Float32.((channelview(img) .- meanC10) ./ stdC10)
    return permutedims(tmp, (3, 2, 1))
end

[DrChainsaw]

Thanks for hitting me back on this. I didn't mean for you to have to read through the horrible code in this repo. It was never meant to be read by anyone :)

Yes, that is a deviation (due to augmentor only having per pixel mappings) but it does unfortunately not explain the accuracy difference. I get more or less the exact same performance when using per channel standardization (I retried it just to be sure). Even though the mean and std are slightly different for each channel, they just don't seem to be different enough to impact the performance.

In case you do try to benchmark that small resnet with Cifar10 I'd be more than happy to know how it went. Thanks again!

[a-r-n-o-l-d]

I have tried some quick test on CIFAR10, without data augmentation and a constant learning rate, but I am far to obtain a "good" accuracy (Test accuracy => 0.83 after 20 epochs).
"I didn't mean for you to have to read through the horrible code in this repo." :) Sure I understand that, but I am learning Julia for a few months only, so I like to have a look at source code from other people.

I notice that your Cutout implementation fills the mask with white noise and not zeros (cutout paper). In addition, the mask size seems to be fixed in original benchmark and not randomly chosen. Maybe it is not a big deal.

By the way, can I borrow your Cutout and RandomCrop implementation? I am trying to understand the source code of Augmentor.jl, but it is far too complex for me.

[DrChainsaw]

Haha, as long as you don't use it to base your judgement of how easy Julia code is to read you can take whatever you want from this repo :)

I tried with zeros as well and that did not have a noticeable effect on the performance compared to what I use here. I think I saw some python repo (maybe the python version of augmentor) which used noise for cutout and thought it was a good idea to keep the batch statistics unit standardized.

Hmm, I need to check the original benchmark again. I could have sworn it looked like the cutout size was randomly rolled each time. I'll try to do a run with fixed size just in case although it probably doesn't matter.

Note that Augmentor has added RCropSize which should do the exact same as my RCrop implementation in a less recursive-horror-story way. I had some varying degrees of luck to get Augmentors augmentbatch to work with the builtins though...

[a-r-n-o-l-d]

Don't worry, I won't judge Julia programming language (after two decades of writing scientific code in R, C, C++ and java, it's a real pleasure to use Julia (and Flux ecosystem)).

Thank you to notify me of the RCropSize existence, it is missing from the Augmentor.jl documentation (I should browse the source code more often).

I also have had some problems with augmentbatch.