training and validation loss nan

[Barfknecht]

First of all I just want to thank you for the great work.
I am having an issue during training, my loss and val_loss is nan, however I am still getting values for accuracy and val_acc. I am training on the PASCAL_VOC 2012 dataset with the segmentation class pngs.

keras 1.2.1 & 2.0.6
tensorflow-gpu 1.2.1
python 3.6.1

[nicolov]

Have you managed to fix this?

…

On 9 August 2017 at 12:00, Barfknecht ***@***.***> wrote: Closed #26 <#26>. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#26 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AKO4B3VTj9VE7wM8TXs_n98e3t6dbBwtks5sWgG6gaJpZM4OyQ7A> .

[Barfknecht]

Yes, the dataset was corrupt somehow. Just re-downloaded and it is fine.

[dimaxano]

@Barfknecht Could you, please, tell what exactly was wrong with your dataset, because I have this problem on my own dataset. But it doesn't look corrupted

[Barfknecht]

@dimaxano Can you give an example of an image?
The issue with mine was that the pixel values of the masks were not valid, so instead of them indicating the class, it had corrupted it and changed it to random ASCII characters.

[dimaxano]

Here is one of my image

[Barfknecht]

@dimaxano Well I can see your problem. So the issue is that the image mask has not been correctly formatted. So if you look in the benchmark_RELEASE/dataset/pngs folder you will see how the masks have been formatted. For example, 2008_000026.png is a mask of a person. That is class 15 of this altered pascal_voc dataset. When you open the png as a matrix, every pixel where the person is located has a pixel value of 15. So the input image is grayscale where the pixel values represent the class associated with it. So I have added one more class to this set, namely tyres. I have then made this my 21st class and when I created the mask, every pixel which is a tyre I set that pixel value to 21.

I hope this helps

[dimaxano]

@Barfknecht Thanks for advice! Now it works

[Barfknecht]

@dimaxano No problem, can you give me some advice on how you trained on your own dataset?

[dimaxano]

@Barfknecht For now, I have some troubles with predicting (#33), but training was quite easy (except this problem with labels ). It is more important to prepare data in a right way