Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reproducing of the paper results #41

Closed
leekanggeun opened this issue Oct 13, 2019 · 6 comments
Closed

Reproducing of the paper results #41

leekanggeun opened this issue Oct 13, 2019 · 6 comments

Comments

@leekanggeun
Copy link

Hello, I have one issue for reproducing the quantitative results of paper.
Even though I set up same parameter with paper, but I couldn’t get paper result for BSD68 with sigma=25 additive white gaussian noise.
So Could you give me the original code to get paper result? Because I want to cite your paper but I couldn’t reproduce exactly..
I mean original code would be training and testing code for BSD400 and BSD68 include dataset.

@alex-krull
Copy link
Contributor

alex-krull commented Oct 14, 2019

Dear @leekanggeun,
Due to the stochastic nature of neural network training, results can vary slightly between individual runs.
So you should not expect to see the precise same numbers.
However, the numbers should definitely not be systematically worse.
Could you please share the numbers you obtained with us?
Thanks!

@leekanggeun
Copy link
Author

leekanggeun commented Oct 14, 2019

Thank you for response.
I set epochs and iteration/epoch as like 100, 300. Accuracy was 26.92 average PSNR. I thought it converges enough to lower bound. I know it's almost impossible to reproduce the same result exactly.
So could you share the avg. PSNR(numerical) for Noise2Void, as shown in Figure 7?

Thanks!!

@alex-krull
Copy link
Contributor

This sounds definitely too low, compared to what we got.
We have unfortunately currently a deadline coming up and cannot look the the problem immediately.
I will get back to you in approx. a week and assemble all the data and code to reproduce the results.
I hope this is still in time for your paper submission?

@leekanggeun
Copy link
Author

I'd appreciate it if you would. To verify my test data set which was generated by gaussian noise(sigma 25), I used bm3d method and got 28.29 while papers bm3d resulr is 28.59. I think that random seed to generate noise will affect to final psnr. Fortunately, I got 27.14 avg PSNR by another parameter of noise2void..but it looks still a little row.. I hope assemble could be completed until November 1. Thank you!!!

@tibuch
Copy link
Collaborator

tibuch commented Nov 1, 2019

Hello @leekanggeun

Thank you for you patience.

Here is the training/validation/test data which we used in our paper and a reproducibility jupyter notebook.

Unfortunately the computation of number of blind-spots in our current release is different from the one used during our paper-experiment time. This is a bug and is fixed on n2v/fix_numPix_computation. If you want to rerun the notebook immediately you would have to install n2v from this branch.

The branch will be merged soon, but we would have to rerun all examples. Additionally we probably add the reproducibility notebook from the zip-file above. In the future reproducing the PSNR numbers should only be a jupyter notebook execution away.

@tibuch
Copy link
Collaborator

tibuch commented Nov 11, 2019

@tibuch tibuch closed this as completed Nov 11, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants