Very Low Accuracy When Using Pretrained Model

[jwfromm]

Description

Pretrained models dont seem to be working well with gluon, specifically datasets build with Dataloader and ImageRecords or ImageFolders.

As an example, here I load the ImageNet validation dataset and feed it into alexnet downloaded from gluon model zoo

ctx = mx.gpu()
batch_size = 64
def transformer(data, label):
    data = mx.image.imresize(data, 224, 224)
    data = mx.nd.transpose(data, (2,0,1))
    data = data.astype(np.float32)
    return data/255, label

test_data = gluon.data.DataLoader(gluon.data.vision.ImageFolderDataset(root="/data2/imagenet/val/", transform=transformer),
                                      batch_size, shuffle=False)

model = gluon.model_zoo.vision.alexnet(pretrained=True, ctx=ctx)

def evaluate_accuracy(data_iterator, net):
    acc = mx.metric.Accuracy()
    for d, l in data_iterator:
        data = d.as_in_context(ctx)
        label = l.as_in_context(ctx)
        output = net(data)
        predictions = nd.argmax(output, axis=1)
        acc.update(preds=predictions, labels=label)
    return acc.get()[1]

evaluate_accuracy(test_data, model, ctx=ctx)
[0.12393999999999999]

The 12% accuracy shows the issue is probably that the transforms used to train the model dont exactly align with the transforms presented in the Gluon tutorial. It would be nice if an example showing how to properly do this using the new gluon functions were added.

Environment info (Required)

Python 3.6

[jwfromm]

The issue turned out to be that both scaling and normalization are needed to match the pytorch transforms.

def transformer(data, label):
    data = mx.image.imresize(data, 256, 256)
    data, _ = mx.image.center_crop(data, (224, 224))
    data = data.astype(np.float32)
    data = data/255
    data = mx.image.color_normalize(data,
                                    mean=mx.nd.array([0.485, 0.456, 0.406]),
                                    std=mx.nd.array([0.229, 0.224, 0.225])) 
    data = mx.nd.transpose(data, (2,0,1))
    return data, label