SIDD Benchmark Issue: I get a PSNR 39.49db rather than 39.89db

[gauenk]

Thank you for the nice code! I use the Uformer-32 model and match the 39.77dB on the validation srgb dataset for SIDD. However, I get a result of 39.49dB from the SIDD server on the benchmark srgb dataset. Would you mind releasing the script you use to create your submission file from the benchmarking data?

[gauenk]

If I use floating-point or rounded values instead of the default "floor" operator implicitly included in video = video.astype(np.uint8) then the PSNR increases to 39.63. I am still at a PSNR of 39.89db and SSIM of 0.958. Any help is appreciated!

[gauenk]

bump

[gauenk]

Is there any update on this?

[gauenk]

With the updated weights from the update ~21 days ago, I see a slight improvement to 39.55 but I still am under the 39.89 SIDD benchmark result.

[madfff]

I got 39.74/0.958 on SIDD benchmark. Maybe you can try my script.

import scipy.io as sio
from skimage import img_as_ubyte

filepath = os.path.join(args.input_dir, 'BenchmarkNoisyBlocksSrgb.mat')
img = sio.loadmat(filepath)
Inoisy = np.float32(np.array(img['BenchmarkNoisyBlocksSrgb']))
Inoisy /=255.
restored = np.zeros_like(Inoisy)
with torch.no_grad():
for i in range(Inoisy.shape[0]):
for k in range(Inoisy.shape[1]):
noisy_patch = torch.from_numpy(Inoisy[i,k,:,:,:]).unsqueeze(0).permute(0,3,1,2).cuda()
restored_patch = model_restoration(noisy_patch)
restored_patch = torch.clamp(restored_patch,0,1).cpu().detach().permute(0, 2, 3, 1).squeeze(0)
restored[i,k,:,:,:] = img_as_ubyte(restored_patch)

        if args.save_images:
            save_file = os.path.join(result_dir_png, '%04d_%02d.png'%(i+1,k+1))
            utils.save_img(save_file, img_as_ubyte(restored_patch))
            # save denoised data

sio.savemat(os.path.join(result_dir_mat, 'SubmitSrgb.mat'), {"DenoisedBlocksSrgb": restored}, do_compression = True)

[gauenk]

Thank you @madfff. I appreciate your comment. Using your script, I have 39.71/0.958 rather than 39.74/0.958. Still, I find it strange we are missing [39.89db - 39.74db =] 0.15 db.

The MPRNet is reported to have 39.71/0.958 and NBNet is reported to have 39.75/0.959. Uformer allegedly beats them, but we are still unable to reproduce the results.

[gauenk]

I am bumping this so it doesn't close from inactivity. I think being unable to reproduce the key result is worth investigating further. I am excited to hear from the authors at some point though I appreciate they are probably super busy.

[ZhendongWang6]

Sorry for the delayed response. I guess you have used inconsistent SIDD dataset. Following previous MIRNet, MPRNet, and so on, we report the result on the validation set of SIDD.

[gauenk]

Thank you for the reply! My validation results match your reported validation results at 39.77 db.

[gauenk]

I am not sure I understand why this comment is closed.

I am able to match your validation results.

I am unable to match your test results.

[gauenk]

Maybe this was an accident -- I can reopen a new issue no problem!

[ZhendongWang6]

Yes. The PSNR 39.89 dB that we report is got on the validation set. It is the same as previous works, so we can compare Uformer with them easily. And we do not give a result on the SIDD benchmark.

[gauenk]

Ah I didn't understand all these results are on the validation set. Thank you for the pointer here -- I am unable to find this detail in your paper.