Low PSNR when evaluate the reproduction model and pretrained model

[hcwang95]

Thanks for providing the detailed code with instructions to train and test.

I am curious if I have any problem with having pretty low PSNR during the evaluation of my trained model and even the pre-trained model downloaded from Google Drive.

The trained model got trained followed by the instruction in README but the PSNR for 1.1x on B100 is only around 29dB. Then I evaluated the pre-trained model and PSNR for 1.1x is also around 29dB.

Would you mind telling me whether the pre-trained model is the model generating the scores in the paper or preliminary model? Thanks a lot!

[hanlinwu]

I found that there may be some bugs in the latest version of the code. I tried to switch the code to the previous version(2e1a266) and got a PSNR value similar to that in the paper.

[hcwang95]

Thanks a lot, @hanlinwu! I will try it soon and update here if it works well.

[hcwang95]

Here is the update. By rolling back to the commit as @hanlinwu pointed out, I successfully achieved around 41dB PSNR on the Set5 validation set. However, when I test the pre-trained model on B100 (which I created myself by Octave rather than Matlab), the PSNR is even lower (around 27dB).

I found that I could not actually reproduce the input image by running the dataset generation script in MATLAB using Octave. @XuecaiHu Would you mind if you could share the B100 dataset that you use for the training or sharing the Matlab version that you run? Thanks a lot!

[reddandelion217]

Did you solve this problem?@hcwang95

[hcwang95]

@reddandelion217 No, I've tried one another training dataset and still cannot reproduce the performance in the paper. Not sure where is the problem.

[reddandelion217]

@hcwang95 After analyzing the file history, I think the problem comes from the code below in file trainer.py, because there is no reason to do this:
i = 1
h, w,_ = pos_mat.size()
while(pos_mat[i][0][0]<= 1e-6 and i<h):
i = i+1

    j = 1
    #pdb.set_trace()
    h, w,_ = pos_mat.size()
    while(pos_mat[0][j][1]<= 1e-6 and j<w):
        j = j+1

    pos_mat_small = pos_mat[0:i,0:j,:]

and the code below in file metardn.py:
local_weight = self.repeat_weight(local_weight,scale_int,x.size(2),x.size(3))

[supercaoO]

@hcwang95 Hello, have you solved the reproduction problem?
I have a similar problem with you. I used the script ./generate_LR_metasr_X1_X4_idealboy.m to prepare the B100 dataset. Then I used the downloaded model_1000.pt to evaluate the PSNR and SSIM results.
What I got at scales 1.1 ~ 2.0 is as follows:

	X1.1	X1.2	X1.3	X1.4	X1.5	X1.6	X1.7	X1.8	X1.9	X2.0
PSNR	27.71	27.73	27.69	27.26	27.98	27.23	27.10	27.02	26.97	28.33
SSIM	0.8412	0.8417	0.8380	0.8223	0.8424	0.8147	0.8086	0.8030	0.7975	0.8398

The PSNR results I got are much lower than that from the paper. @XuecaiHu Could you please help solve the problem? Thank you!

[henrycliang]

@hcwang95 Can you provide some details about how you solve the low-PSNR problem? I meet the same problem with you. I feel confused about @hanlinwu 's comment -- on the preview version

[XuecaiHu]

@supercaoO can you check the output of the h_project_coord and w_project_coord？

[XuecaiHu]

and show me the results @supercaoO

[supercaoO]

@XuecaiHu Thanks for your reply. I have emailed you (xuecai.hu@cripac.ia.ac.cn) the results.

[supercaoO]

@liangheng96 Hello. After cloning the repository, maybe you can try git reset --hard 2e1a266832ab3a2dd855c98c34387c47a4ebec01 to switch it to the previous version of the repository. I think that is what @hanlinwu means.

[henrycliang]

@supercaoO Thank you! I know what @hanlinwu means by now. Now I swith to the previous version, and then I can get the similar results to the paper. Thank you very much!

[henrycliang]

I think @reddandelion217 's comment is right. I delete these codes and then get the right PSNR.

@hcwang95 After analyzing the file history, I think the problem comes from the code below in file trainer.py, because there is no reason to do this:
i = 1
h, w,_ = pos_mat.size()
while(pos_mat[i][0][0]<= 1e-6 and i<h):
i = i+1

    j = 1
    #pdb.set_trace()
    h, w,_ = pos_mat.size()
    while(pos_mat[0][j][1]<= 1e-6 and j<w):
        j = j+1

    pos_mat_small = pos_mat[0:i,0:j,:]

and the code below in file metardn.py:
local_weight = self.repeat_weight(local_weight,scale_int,x.size(2),x.size(3))

[LILIXIYA]

Hi, do you see any errors in meta.py, saying 'ValueError: only one element tensors can be converted to Python scalars'? Thanks!@hcwang95

[dflateau]

Any updates on this?