NaN loss for graph classification

[chenz97]

Hi, thanks for the code. When I ran the graph classification code by make test_phase=1 save_dir=save data_type='S5CONS', I found that the loss is always NaN both for training and validation, and the performance figures didn't change. Do you have any idea why this might be happening? Thank you!

[ilibarra]

Dear dmis-lab team,

Thank you for providing the code and environment to run your hierarchical graph model. I am getting the same nan loss errors as described above. Because I cannot find a description on how to solve it. I am here providing more details.

Currently using numpy 1.11.0 tensorflow 1.19.5 and CPUs

These are the current outputs

 EPOCH 1 EVAL ALL
loss : nan accuracy : 0.4082 hit ratio : 0.5714 pred_rate : [1.0, 0.0, 0.0] macro f1 : 0.1932 micro f1 : 0.4082 expected return : -0.0009
03/27/2021 02:29:09 PM: EPOCH 11 TRAIN ALL 
loss : nan accuracy : 0.3177 hit ratio : 0.4822 pred_rate : [1.0, 0.0, 0.0] macro f1 : 0.1607 micro f1 : 0.3177 expected return : 0.0001
03/27/2021 02:29:09 PM:

Is there a conda environment or requirements.txt file that could assure this behavior does not occur, hopefully to avoid checking packages one by one?

Thank you for any input.

[ilibarra]

It seems that prob is nan during the first and following epochs, regardless of x, y and labels properly defined. Then loss nan as well.

Any way to troubleshoot / check if y probs are initialized as zero? Thanks.

In graph_trainer.py, line 69

_, loss, pred, prob = self.sess.run([self.model.train_step, self.model.cross_entropy,
                            self.model.prediction, self.model.prob],
                             feed_dict=feed_dict)