SQAD: Automatic Smartphone Camera Quality Assessment and Benchmarking

Official webpage for ICCV'23 paper "SQAD: Automatic Smartphone Camera Quality Assessment and Benchmarking"

Dataset download: SQAD

• There are three folders contained: train, test, and sample_crop16. Follow the instructions below to prepare the data for training and testing.

You can download our pre-trained models from this link: ResNet-based / MANIQA-based

• Here are MANIQA-based models with all 6 quality factors, and ResNet-based models for resolution on cross-camera prediction task. Please refer to Table 4 in the paper for camera set division.

• Feel free to explore on cross-camera prediction task since there is still high potential to improve further.

Baselines

Here we provide some baselines from sota MANIQA model for testing on sample_crop16 subset and some randomly generated test set from test folder. Within one image, the information levels (i.e. edge details, color vividness) for different parts usually vary, there would be fluctuations for the evaluation metrics if you apply your own cropping.

Results evaluated with sample_crop16 subset:

Metrics	Resolution	Color Accuracy	Noise	DR	PSF	Aliasing
SROCC	0.9292	0.9475	0.9336	0.9546	0.9424	0.9172
PLCC	0.9460	0.9592	0.9784	0.9694	0.9823	0.9812

Results evaluated with randomly cropped subset from test folder:

Metrics	Resolution	Color Accuracy	Noise	DR	PSF	Aliasing
SROCC	0.9475	0.9242	0.8756	0.9328	0.9117	0.8940
PLCC	0.9479	0.9414	0.9373	0.9332	0.9704	0.9688

• For cross-camera prediction, please refer to Ablations section in our paper for more details. Here we provide results for multi-crop (16×) predictions .

Models	Metrics	SET 1	SET 2	SET 3	SET 4	SET 5	Overall
ResNet50	SROCC	0.5747	0.5590	0.6700	0.4017	0.8481	0.6212
ResNet50	PLCC	0.8161	0.4244	0.6837	0.6539	0.6160	0.6075
MANIQA	SROCC	-	-	-	-	-	-
MANIQA	PLCC	0.6585	0.4087	0.8354	0.6324	0.7529	0.4486

Model training

For the quality factor regression with MANIQA model, please follow the guidelines provided by CVPRW 2022: MANIQA to set up the environment. Thanks for their great works on image quality assessment.

For ResNet50-based backbone, there is no specific requirement for the environment settings. Just make sure you have installed pytorch and Pillow properly. If there are some package related issues, follow the error instructions.

Dataset Preparation

• Please download the SQAD dataset, unzip and put all files into the data folder in the following structure:

SQAD
|—— data
|	|—— processed
|	|—— train
|	|	|—— 01_ASUS_Z00AD
|	|	|	|—— P_20000102_085109.jpg & ...
|	|	|—— 02_...
|	|—— test
|	|	|—— 01_ASUS_Z00AD
|	|	|—— 02_...
|	|—— sample_crop16
|—— prepare_data.py
|—— camera_ground_truth.json
|—— maniqa_plugin
|—— resnet_

To get the image crops for training, simply run:

python prepare_data.py

Or use the following command to specify the crop number and crop size (based on backbone network) per-image. You can also add --crop_test_dataset flag to create a randomly cropped test set for model evaluations.

python prepare_data.py --crop_num 32 --crop_size 240 --crop_test_dataset

Then, the cropped images and corresponding label files for training are stored in ./data/processed.

For model testing, we have provided a pre-cropped image sets in the folder sample_crop16, which is extracted from the test folder. But it is free to use --crop_test_dataset flag to create your own test test as well.

Train with MANIQA

Please clone the MANIQA project and replace the original train_maniqa.py and inference.py by the files with the same name in maniqa_plugin folder.

git clone https://github.com/IIGROUP/MANIQA.git

Then, add camera29.py in maniqa_plugin/data into the original MANIQA data folder. Now you can modify the configurations in train_maniqa.py Line136-141 and inference.py Line75-78 to play with the model training and quality factor inference.

• The "factor" element in config uses the same key name with camera_ground_truth.json, they are: resolution, color_acc, noise, dr, psf, aliasing.

• Download the pre-trained models and put them into the save path you specified in inference.py for direct evaluation.

Train with ResNet backbone

To train the regression model with ResNet50 backbone, run train_regress.py file in the resnet_ folder with different quality factors:

python train_regress.py --factor dr(resolution/color_acc/...) --batch_size 64 --train_epochs 200 --save_eval_every 1

• To utilize the entire test set, one can also apply the random cropping during DataLoader and specify different image crop sizes and desired crop numbers. We provide a FullImageDataset class in CameraDataset.py for exploring. Note that the running speed for this test would be much slower than the pre-cropped setting, since the dataloader needs to load the full-size image first, which is slightly time-consuming for some high quality photos captured by high-end mobile phones.