Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

training demo doesn't work #2

Open
cslxiao opened this issue Sep 29, 2016 · 2 comments
Open

training demo doesn't work #2

cslxiao opened this issue Sep 29, 2016 · 2 comments

Comments

@cslxiao
Copy link

cslxiao commented Sep 29, 2016

When I run the training demo LRR4xTrainVGG16Pascal, it goes fine at the beginning. However, after several iterations, I got the following output:

obj_dil_seg32x: NaN obj_ero_seg32x: NaN objective_32x: NaN

What's wrong with it? I didn't change anything expect for the directories of the dataset.

@golnazghiasi
Copy link
Owner

I'm not sure why this is happening. I remember I had the exact same problem with the training code of fcn-matconvet (https://github.com/vlfeat/matconvnet-fcn). I could resolve the problem by using a smaller learning rate for just first few epochs. Maybe you can also try this.
Can I ask what version of Matconvnet do you use? Also what is your gpu?

@cslxiao
Copy link
Author

cslxiao commented Sep 30, 2016

Thanks @golnazghiasi .
I used the latest version of Matconvnet (master branch) on the github.
And I am using Titan X.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants