about the eval result, little smaller than the paper's, why?

[robotzheng]

[root@A01-R04-I220-17 TTSR]# sh eval.sh
[2020-06-23 15:19:04,069] - [trainer.py file line:48] - INFO: load_model_path: ./TTSR-rec.pt
[2020-06-23 15:19:04,143] - [trainer.py file line:121] - INFO: Epoch 0 evaluation process...
[2020-06-23 15:20:40,578] - [trainer.py file line:150] - INFO: Ref PSNR (now): 26.991 SSIM (now): 0.8003
[2020-06-23 15:20:40,580] - [trainer.py file line:158] - INFO: Ref PSNR (max): 26.991 (0) SSIM (max): 0.8003 (0)
[2020-06-23 15:20:40,580] - [trainer.py file line:160] - INFO: Evaluation over.
[root@A01-R04-I220-17 TTSR]# sh eval.sh
[2020-06-23 15:16:24,885] - [trainer.py file line:48] - INFO: load_model_path: ./TTSR.pt
[2020-06-23 15:16:24,948] - [trainer.py file line:121] - INFO: Epoch 0 evaluation process...
[2020-06-23 15:17:21,531] - [trainer.py file line:150] - INFO: Ref PSNR (now): 25.402 SSIM (now): 0.7600
[2020-06-23 15:17:21,532] - [trainer.py file line:158] - INFO: Ref PSNR (max): 25.402 (0) SSIM (max): 0.7600 (0)
[2020-06-23 15:17:21,532] - [trainer.py file line:160] - INFO: Evaluation over.

[FuzhiYang]

Hi, these two results are from only using L1 reference image in CUFED5 test set, which is consistent in Table 4 in our paper.
However, there are totoally 5 reference images with different similarity levels for each input test image. We stitch 5 reference images together offline as a whole ref to get the results in Table 1, which is the same setup with CVPR 2019 SRNTT paper.

[FuzhiYang]

Please let me know whether there are additional questions about the quantitative results. If no, I'll closed this issue this day~

[scutlrr]

Hi, these two results are from only using L1 reference image in CUFED5 test set, which is consistent in Table 4 in our paper.
However, there are totoally 5 reference images with different similarity levels for each input test image. We stitch 5 reference images together offline as a whole ref to get the results in Table 1, which is the same setup with CVPR 2019 SRNTT paper.

Whether the eval result on other datasets is also taken 5 reference images together offline as a whole ref to calculate metric？ Could you please give some tips for the “stitch” or where can I found it in SRNTT's code？

[FuzhiYang]

Since CUFED5 test dataset has 5 reference images, so we put them together as a whole ref, which is the same setup as SRNTT. Other datasets does not have 5 references.

There are no detailed description about how to put them together in SRNTT paper. We just first complete each ref to 500x500 resolution with black background and then stich them in one column, such that the whole ref is 2500x500 resolution.

[scutlrr]

Since CUFED5 test dataset has 5 reference images, so we put them together as a whole ref, which is the same setup as SRNTT. Other datasets does not have 5 references.

There are no detailed description about how to put them together in SRNTT paper. We just first complete each ref to 500x500 resolution with black background and then stich them in one column, such that the whole ref is 2500x500 resolution.

Sorry, I forgot other datasets do not have 5 references. Thank you for your reply.

[scutlrr]

I have stitched all ref image into a 2500x500 resolution image, but when I eval the model using 1080ti, CUDA out of memory, can you tell me how big is your GPU memory?