Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
The EuroSAT paper, a geo-referenced aerial/satellite image dataset of 27,000 images categorized into 10 different classes is said to achieve 98.57% classification accuracy using CNNs. More specifically, using ResNet50, it achieves 96.37% accuracy using a 90/10 train/test split. Using ML.NET Image Classification API as well as Model Builder achieves 99%+ accuracy while training. However, when evaluating the model, both with and without cross validation, accuracy drops between 61-69% using only the CPU and 59% using the GPU. See performance comparisons in table below.
Source code / logs
The source code is at the following repo: https://github.com/luisquintanilla/EuroSATTrainSample
Yeah, I tried that same image-set with Model Builder and trained with GPU.
I also attach the log info captured from Visual Studio output that you can download from here:
@codemzs here's the code doing train/test split.
Below is the file I used to read the file paths and labels. To test, you can change the parent directory
Minor: Cross-Validation won't be used in ModelBuilder as the dataset has 27,000 rows. Standard train-validate mode will be used.
We should look into why the Image Classifier API isn't hitting the expected numbers. Perhaps we can compare the training script from the paper to our implementation. As a comparison point, the DNNImageFeaturizer style gets a bit over 94% accuracy taking 18min on CPU for its first model (then continues sweeping). Hence can replicate similar (but lower) scores as the paper implying the dataset is not at fault.
Here are the splits I was using:
Hi @luisquintanilla ,
As I suspected your comparison was not apples to apples with regards to the EuroSAT paper.
If you do the above then you will get an accuracy of 97.1833333333333% (roughly one percentage point higher than the best accuracy reported in EuroSAT paper using ResNet50), please see the attached logs - EuroSAT_90_10_split_200_epochs_shuffle.txt
I'm closing this issue because there is no issue with Image Classification API but you were getting low accuracy because you were not using the API correctly to make fair comparisons with the paper.