You may find the python requirements in the file requirements.txt
.
The pre-trained model-weights MNIST
, FMNIST
, CIFAR10
and SVHN
are provided in the directory models
.
The pre-trained model weights for Imagenette
are directly loaded from the torchvision
package.
In order to run experiments for the Imagenette
dataset you will first need to download the dataset from https://github.com/fastai/imagenette and save if in the directory /data/imagenette
.
Proceed with the following steps:
- Run the python script
MAGDiff_feature_generation.py
with an input parameter combination from the following parameter grid:
{
-idn : [MNIST, FMNIST, CIFAR10, SVHN]
-wgt : [dense3, dense2_1, dense2, fc]
-sd : [1.0, 0.5, 0.25]
-sfn : [gaussian_noise, gaussian_blur, image_shift]
-sfi : [I, II, III, IV, V, VI]
}
- The
-wgt
parameter controls which layer of the model is considered. ForSVHN
andCIFAR10
, onlyfc
can be used. ForMNIST
andFMNIST
:dense3
corresponds tol_-1
,dense2_1
tol_-2
anddense2
tol_-3
in the notation of the paper. - The
-sd
parameter controls thedelta
. - The
-sfn
parameter corresponds to the shift function name. - The
-sfi
parameter controls the shift intensity. (If you want to generate plots later on you must run each parameter combination for all shift intensities.)
Please first run this script for all parameters that you are interested in, for example:
python MAGDiff_feature_generation.py -idn MNIST -wgt dense3 -sd 1.0 -sfn gaussian_noise -sfi VI
This will save the resulting MAGDiff features in the directory /results/MAGDiff_features
.
- Next, run the script
MAGDiff_evaluation.py
with parameters from the following parameter grid (only the ones for which you've already completed the previous step):
{
-idn : [MNIST, FMNIST, CIFAR10, SVHN]
-wgt : [dense3, dense2_1, dense2, fc]
-sd : [1.0, 0.5, 0.25]
-sfn : [gaussian_noise, gaussian_blur, image_shift, ko-shift]
}
This will execute the statistical tests and save the results in the directory /results/tables
.
-
Once this is done, run the script
collecting_results.py
. This will collect all previously generated results in single files which will be saved in/results/tables/collected_results/
. -
Finally, run the script
MAGDiff_plots.py
. This will create the plots, as in the paper, of all the previously generated results. the plots will be saved in the directoryresults/figures
.
Note: In the tables, the PV-BL
entries corresponds to the results for the baseline, called CV
in the paper, and MN
corresponds to the MAGDiff matrix norm.