Weird training accuracy curve

[gongliyu]

Hi,

I observed that the training accuracy is very low all the time during the training process. The values is around 50%. Although validation accuracy reaches around 83%, this is still weird. I tried Kipf's GCN code, it can reach above 90% training accuracy, which is much normal.

Does anybody notice this?

[PetarV-]

Hi Liyu,

This is caused by the strong dropout on the alpha_ij values. It causes the network to mispredict more often during training, as the graph neighbourhoods get quite variable; however this makes it less prone to overfitting on small training datasets such as Cora.

(If you were to turn off dropout and evaluate the model on the training set at the end of each epoch, you would probably get much more normal progression of training accuracies.)

Hope that helps!

Thanks,
Petar

[gongliyu]

Hi Petar,

Got it. Thank you very much for the explanation.

Best,
Liyu