-
Notifications
You must be signed in to change notification settings - Fork 530
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
using --mode=visualize after trained the model, but I got the wrong pred_x.png #35
Comments
ok. I've seen the same problem in the previous issues. I'll try to add the batchsize up to 16,and try again. And I wish it will work. |
@ChinaYi Maybe you can use "logs/images/Image_Cmaped.ipynb" to visualize your output. |
@yeshenlin thanks, I've tried this, but did not work, actually, it turns an all-black png to an all-blue one. |
@ChinaYi Is it possible that the model hasn't been trained for enough epochs? |
@shekkizh thanks for your help. My iteration times are set to 1e+5, and the best valid_loss is 0.566078. Results are similiar to groud_truth, but not enough comparing to your report. Is there any possible reasons? |
Hi @ChinaYi : I am experiencing the same problem as you previously got. After training for 100K (default) iterations, the prediction image is just black. Did you do anything special to fix this issue? Step: 99800, Train_loss:3.27614 |
@placentian it seems that your loss has been fluctuated with the iteration times and haven't reached the bottom. This is not a good training process. You'd better enlarge the batch size and try to reduce to learning rate. I have been suffered from the same issues like you, you got black picture since you have a large loss. cheer up! |
Thanks a lot for your encouragement, @ChinaYi . Do I also need to change the iteration to some other numbers (like 200K or 500K)? What kind of reasonable learning rate and batch size would you recommend by any chance? |
I got better results after reducing the learingrate but they are not as good as @shekkizh 's results. Dont know what else we should try. |
@placentian can you plot you loss curve? Or paste the Train_loss for each iteration |
Maybe I can explain why we got black outputs. |
the output file pred_x.png has no color except black.
The text was updated successfully, but these errors were encountered: