Use tf.softmax_cross_entropy_with_logits to calculate loss #181
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@elezar Thanks for your contribution. I ran the cluttered-mnist example again. Within 500 epochs, I did not see the 'nan' loss. I speculate the previous problem is due to negative log likelihood in cross entropy. In some case, when the probability of true category is close to 0, the numerical loss would be 'nan'. It could be caused by error in labeling or model divergence. I guess the build-in loss module has some scheme to avoid it? Anyway, I think the problem has been solved for this example. |
|
@martinwicke could you have a look again? (@denny1108 it would be great if you could also just confirm that things still look good from your side) |
|
Looks good, thanks! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR closes #166
When manually implementing the cross entropy loss, the training loss becomes
nanafter a number of iterations (usually around 90 or so epochs for the cluttered MNIST example). Switching to the built-insoftmax_cross_entropy_with_logitsallows the network to train stably.@denny1108 could you also just confirm that this solves the problem?