built-in image-classification algorithm: validation accuracy doesn't match reported val accuracy

[Mahi-Mai]

System Information

• Framework (e.g. TensorFlow) / Algorithm (e.g. KMeans): conda_mxnet_p36 (for evaluation)
• Python Version: 3.6

Describe the problem

I'm using the built-in image classification algorithm supplied by AWS Sagemaker.

I have a binary image classification problem that I've launched from both the console and Jupyter Notebook instances. I define training and validation data channels, supplying rec files for each. Training runs and completes. The graphs on the training job page report a high validation accuracy at the end of the training job's lifetime- over 90%.

To better understand this accuracy, I open a Jupyter notebook, reload the model using sagemaker.estimator.Estimator.attach() and deploy it to an endpoint. A bring in my folder of validation images- the same ones used to create the validation channel rec file- from S3 and add them to a list. I loop over this list and evaluate each image using the code from Image-classification-transfer-learning-highlevel.ipynb, collecting the results.

Here's the code for that.

predictions = []
# test_files is a list of all the image paths in the validation set

for file_name in test_files:
    with open(file_name, 'rb') as f:
        payload = f.read()
        payload = bytearray(payload)

    working_model.content_type = 'application/x-image'
    result = json.loads(working_model.predict(payload))

    # the result will output the probabilities for all classes
    # find the class with maximum probability and print the class index
    index = np.argmax(result)
    object_categories = ['CATEGORY_1', 'CATEGORY_2']
    
    file = file_name.split('/')[-1]
    
    predictions.append([file, dataset, result, index, object_categories[index]])

When I evaluate these predictions against the ground truth values I get an accuracy around 50%.

What could be the reason for this discrepancy? Am I misunderstanding the graphs reporting validation accuracy on the training job page? (I've attached a sample graph.)

There's a lot of moving pieces to this question, so please let me know what additional, specific information would be useful in answering.

Thanks!

[Mahi-Mai]

The same dataset has been used to create a Keras model in a Jupyter Notebook. The outcome of that model was a high reported validation accuracy. Evaluating the validation data separately on the model showed a matching validation accuracy.

I believe this implies there's nothing inherently wrong with the data and how it's been setup.

[Mahi-Mai]

If there's a better channel through AWS to get this problem solved, please let me know. AWS Support services weren't able to help.

[gurumurthys]

Can you please provide the parameters you used for training and in particular if you used the resize parameter?

[Mahi-Mai]

Thank you, @gurumurthys ! We didn't use the resize parameter. I've attached screenshots of the training channel, validation channel, and the hyperparameters from the training job.

It's disconcerting that the user can't define the number of samples or batch size for the validation set. There isn't additional splitting going on, is there? The point of defining a training and validation channel is to predefine the split...

[Mahi-Mai]

Also, these are greyscale images, which mean they're single channel. .... hmmm...

[Mahi-Mai]

@gurumurthys

For kicks I re-ran the training job, defining image shape as 1, 256, 256. It failed, saying that transfer learning only takes 3 channel images.

Should I take this to mean that Sagemaker is converting my single channel grayscale image to a 3 channel image by replicating the single channel three times whenever I define image shape as 3, 256, 256? Or what's happening in the background, there?

[gurumurthys]

Yes, it is converting your grayscale image into 3-channel images based on image_shape. Can you provide details about the raw image size? Since you have not specified any data augmentation and no resize parameter, the center region is cropped for training and validation. For inference through the endpoint, the image is resized to image shape parameter.

[Mahi-Mai]

@gurumurthys

The raw image shape matches the one defined in the hyperparameters: 3, 256, 256

[Mahi-Mai]

Good news! I was discussing some of the comments made in the Git issue with someone, and after a little digging we discovered that the raw images are, in fact, all different sorts of sizes! As an experiment, I took the center 256x256 pixel crop of each image and used that for our manual validation, and the accuracy was much better and closer to what we'd expect.

We're confident this is the root of our issues. Thank you for all your help!