Cannot reproduce Dygie++ results #9
Description
Hi @schmidek
Thank you a lot for your contribution! We tried to launch your project on KnowledgeNet Dataset (this version: https://github.com/schmidek/dygiepp/tree/multitask/dygie) and encounter with the following issue: the training does not yield the expected high scores and a NaN or Inf warning is constantly raised during training.
2020-11-17 11:10:03,030 - INFO - allennlp.training.trainer - Training
0%| | 0/1074 [00:00<?, ?it/s]2020-11-17 11:10:04,066 - WARNING - allennlp.training.util - Metrics with names beginning with "_" will not be logged to the tqdm progress bar.
rel_precision: 0.0000, rel_recall: 0.0000, rel_f1: 0.0000, rel_span_recall: 0.0057, loss: 3.1743 ||: 9%|9 | 99/1074 [01:05<09:49, 1.65it/s]2020-11-17 11:11:10,009 - WARNING - root - NaN or Inf found in input tensor.
2020-11-17 11:11:10,009 - WARNING - root - NaN or Inf found in input tensor.
2020-11-17 11:11:10,026 - WARNING - root - NaN or Inf found in input tensor.
2020-11-17 11:11:10,026 - WARNING - root - NaN or Inf found in input tensor.
2020-11-17 11:11:10,029 - WARNING - root - NaN or Inf found in input tensor.
2020-11-17 11:11:10,029 - WARNING - root - NaN or Inf found in input tensor.
2020-11-17 11:11:10,038 - WARNING - root - NaN or Inf found in input tensor.
2020-11-17 11:11:10,038 - WARNING - root - NaN or Inf found in input tensor.
2020-11-17 11:11:10,041 - WARNING - root - NaN or Inf found in input tensor.
2020-11-17 11:11:10,041 - WARNING - root - NaN or Inf found in input tensor.
rel_precision: 0.0000, rel_recall: 0.0007, rel_f1: 0.0000, rel_span_recall: 0.0060, loss: 2.1448 ||: 19%|#8 | 199/1074 [02:09<08:33, 1.70it/s]2020-11-17 11:12:13,805 - WARNING - root - NaN or Inf found in input tensor.
2020-11-17 11:12:13,806 - WARNING - root - NaN or Inf found in input tensor.
2020-11-17 11:12:13,818 - WARNING - root - NaN or Inf found in input tensor.
2020-11-17 11:12:13,818 - WARNING - root - NaN or Inf found in input tensor.
2020-11-17 11:12:13,820 - WARNING - root - NaN or Inf found in input tensor.
2020-11-17 11:12:13,820 - WARNING - root - NaN or Inf found in input tensor.
2020-11-17 11:12:13,827 - WARNING - root - NaN or Inf found in input tensor.
2020-11-17 11:12:13,827 - WARNING - root - NaN or Inf found in input tensor.
2020-11-17 11:12:13,829 - WARNING - root - NaN or Inf found in input tensor.
2020-11-17 11:12:13,830 - WARNING - root - NaN or Inf found in input tensor.
rel_precision: 0.0000, rel_recall: 0.0005, rel_f1: 0.0000, rel_span_recall: 0.0054, loss: 1.7052 ||: 26%|##5 | 276/1074 [02:58<08:36, 1.54it/s]
...
The setup is:
- the data was loaded from the link mentioned above, and used as-is
- requirements are installed (including SciBERT)
- Python 3.7
- config is used
We run the following command: .\scripts\train\train_kn.sh gpu_id.
This NaN warning appears a couple of times in each epoch. The inter-evaluation also does not improve over the epochs.
Our guess is that the problem might be in the training configs, since the NaN issue might be caused by the older version of allennlp and its tensorboard logging (allenai/allennlp#3116). Did you use the same config as mentioned above for the training of Dygie++ for the KnowledgeNet leaderboard?
Could you help us please? Did we miss anything in the preprocessing that enables successful training?
Evaluation after the first epoch
2020-11-17 11:29:18,090 - INFO - allennlp.training.trainer - Validating
rel_precision: 0.0000, rel_recall: 0.0000, rel_f1: 0.0000, rel_span_recall: 0.0000, loss: 0.2731 ||: 100%|##########| 1022/1022 [01:01<00:00, 16.75it/s]
2020-11-17 11:30:19,115 - INFO - allennlp.training.tensorboard_writer - Training | Validation
2020-11-17 11:30:19,116 - INFO - allennlp.training.tensorboard_writer - _arg_id_precision | 0.000 | 0.000
2020-11-17 11:30:19,117 - INFO - allennlp.training.tensorboard_writer - _trig_class_precision | 0.000 | 0.000
2020-11-17 11:30:19,118 - INFO - allennlp.training.tensorboard_writer - _arg_class_f1 | 0.000 | 0.000
2020-11-17 11:30:19,118 - INFO - allennlp.training.tensorboard_writer - _arg_class_precision | 0.000 | 0.000
2020-11-17 11:30:19,119 - INFO - allennlp.training.tensorboard_writer - gpu_0_memory_MB | 7333.000 | N/A
2020-11-17 11:30:19,119 - INFO - allennlp.training.tensorboard_writer - _trig_id_precision | 0.000 | 0.000
2020-11-17 11:30:19,119 - INFO - allennlp.training.tensorboard_writer - _arg_class_recall | 0.000 | 0.000
2020-11-17 11:30:19,120 - INFO - allennlp.training.tensorboard_writer - gpu_4_memory_MB | 4.000 | N/A
2020-11-17 11:30:19,120 - INFO - allennlp.training.tensorboard_writer - _ner_precision | 0.000 | 0.000
2020-11-17 11:30:19,120 - INFO - allennlp.training.tensorboard_writer - cpu_memory_MB | 3344.232 | N/A
2020-11-17 11:30:19,121 - INFO - allennlp.training.tensorboard_writer - _coref_mention_recall | 0.000 | 0.000
2020-11-17 11:30:19,121 - INFO - allennlp.training.tensorboard_writer - _trig_id_recall | 0.000 | 0.000
2020-11-17 11:30:19,122 - INFO - allennlp.training.tensorboard_writer - gpu_2_memory_MB | 10289.000 | N/A
2020-11-17 11:30:19,122 - INFO - allennlp.training.tensorboard_writer - _trig_id_f1 | 0.000 | 0.000
2020-11-17 11:30:19,123 - INFO - allennlp.training.tensorboard_writer - gpu_5_memory_MB | 2158.000 | N/A
2020-11-17 11:30:19,123 - INFO - allennlp.training.tensorboard_writer - gpu_6_memory_MB | 10289.000 | N/A
2020-11-17 11:30:19,123 - INFO - allennlp.training.tensorboard_writer - gpu_7_memory_MB | 7649.000 | N/A
2020-11-17 11:30:19,123 - INFO - allennlp.training.tensorboard_writer - _coref_recall | 0.000 | 0.000
2020-11-17 11:30:19,124 - INFO - allennlp.training.tensorboard_writer - _trig_class_recall | 0.000 | 0.000
2020-11-17 11:30:19,124 - INFO - allennlp.training.tensorboard_writer - _ner_recall | 0.000 | 0.000
2020-11-17 11:30:19,125 - INFO - allennlp.training.tensorboard_writer - _coref_precision | 0.000 | 0.000
2020-11-17 11:30:19,125 - INFO - allennlp.training.tensorboard_writer - rel_recall | 0.000 | 0.000
2020-11-17 11:30:19,126 - INFO - allennlp.training.tensorboard_writer - _args_multiple | 0.000 | 0.000
2020-11-17 11:30:19,126 - INFO - allennlp.training.tensorboard_writer - rel_precision | 0.000 | 0.000
2020-11-17 11:30:19,127 - INFO - allennlp.training.tensorboard_writer - _trig_class_f1 | 0.000 | 0.000
2020-11-17 11:30:19,127 - INFO - allennlp.training.tensorboard_writer - _coref_f1 | 0.000 | 0.000
2020-11-17 11:30:19,128 - INFO - allennlp.training.tensorboard_writer - _ner_f1 | 0.000 | 0.000
2020-11-17 11:30:19,128 - INFO - allennlp.training.tensorboard_writer - gpu_3_memory_MB | 9653.000 | N/A
2020-11-17 11:30:19,128 - INFO - allennlp.training.tensorboard_writer - loss | 0.688 | 0.273
2020-11-17 11:30:19,129 - INFO - allennlp.training.tensorboard_writer - rel_span_recall | 0.002 | 0.000
2020-11-17 11:30:19,129 - INFO - allennlp.training.tensorboard_writer - rel_f1 | 0.000 | 0.000
2020-11-17 11:30:19,130 - INFO - allennlp.training.tensorboard_writer - _arg_id_recall | 0.000 | 0.000
2020-11-17 11:30:19,130 - INFO - allennlp.training.tensorboard_writer - _arg_id_f1 | 0.000 | 0.000
2020-11-17 11:30:19,130 - INFO - allennlp.training.tensorboard_writer - gpu_1_memory_MB | 9653.000 | N/A
2020-11-17 11:30:21,546 - INFO - allennlp.training.checkpointer - Best validation performance so far. Copying weights to './models/kn/best.th'.
2020-11-17 11:30:22,486 - INFO - allennlp.training.trainer - Epoch duration: 0:12:33.792339
2020-11-17 11:30:22,488 - INFO - allennlp.training.trainer - Estimated training time remaining: 10:15:35
2020-11-17 11:30:22,488 - INFO - allennlp.training.trainer - Epoch 1/49
Evaluation after epoch 22 (early-stopping)
rel_precision: 0.0001, rel_recall: 0.0269, rel_f1: 0.0001, rel_span_recall: 0.0370, loss: 0.4834 ||: 100%|##########|
1022/1022 [00:57<00:00, 17.70it/s]
2020-11-17 15:06:06,074 - INFO - allennlp.training.trainer - Ran out of patience. Stopping training.
2020-11-17 15:06:06,076 - INFO - allennlp.training.checkpointer - loading best weights
2020-11-17 15:06:06,349 - INFO - allennlp.commands.train - To evaluate on the test set after training, pass the 'evaluate_on_test' flag, or use the 'allennlp evaluate' command.
2020-11-17 15:06:06,350 - INFO - allennlp.models.archival - archiving weights and vocabulary to ./models/kn/model.tar.gz
2020-11-17 15:06:30,631 - INFO - allennlp.common.util - Metrics: {
"best_epoch": 8,
"peak_cpu_memory_MB": 5457.956,
"peak_gpu_0_memory_MB": 7333,
"peak_gpu_1_memory_MB": 9653,
"peak_gpu_2_memory_MB": 10289,
"peak_gpu_3_memory_MB": 9653,
"peak_gpu_4_memory_MB": 10439,
"peak_gpu_5_memory_MB": 1079,
"peak_gpu_6_memory_MB": 10289,
"peak_gpu_7_memory_MB": 7649,
"training_duration": "3:17:07.685956",
"training_start_epoch": 0,
"training_epochs": 22,
"epoch": 22,
"training__coref_precision": 0.0,
"training__coref_recall": 0.0,
"training__coref_f1": 0.0,
"training__coref_mention_recall": 0.0,
"training__ner_precision": 0.0,
"training__ner_recall": 0.0,
"training__ner_f1": 0.0,
"training_rel_precision": 3.5733284070843595e-05,
"training_rel_recall": 0.013081574725410341,
"training_rel_f1": 7.127188374614434e-05,
"training_rel_span_recall": 0.016413673947920523,
"training__trig_id_precision": 0,
"training__trig_id_recall": 0,
"training__trig_id_f1": 0,
"training__trig_class_precision": 0,
"training__trig_class_recall": 0,
"training__trig_class_f1": 0,
"training__arg_id_precision": 0,
"training__arg_id_recall": 0,
"training__arg_id_f1": 0,
"training__arg_class_precision": 0,
"training__arg_class_recall": 0,
"training__arg_class_f1": 0,
"training__args_multiple": 0,
"training_loss": 0.010883575075255758,
"training_cpu_memory_MB": 5334.428,
"training_gpu_0_memory_MB": 7333,
"training_gpu_1_memory_MB": 9653,
"training_gpu_2_memory_MB": 10289,
"training_gpu_3_memory_MB": 9653,
"training_gpu_4_memory_MB": 10439,
"training_gpu_5_memory_MB": 1079,
"training_gpu_6_memory_MB": 10289,
"training_gpu_7_memory_MB": 7649,
"validation__coref_precision": 0.0,
"validation__coref_recall": 0.0,
"validation__coref_f1": 0.0,
"validation__coref_mention_recall": 0.0,
"validation__ner_precision": 0.0,
"validation__ner_recall": 0.0,
"validation__ner_f1": 0.0,
"validation_rel_precision": 5.676047081316843e-05,
"validation_rel_recall": 0.02132435465768799,
"validation_rel_f1": 0.0001132195774526191,
"validation_rel_span_recall": 0.0306771417882529,
"validation__trig_id_precision": 0,
"validation__trig_id_recall": 0,
"validation__trig_id_f1": 0,
"validation__trig_class_precision": 0,
"validation__trig_class_recall": 0,
"validation__trig_class_f1": 0,
"validation__arg_id_precision": 0,
"validation__arg_id_recall": 0,
"validation__arg_id_f1": 0,
"validation__arg_class_precision": 0,
"validation__arg_class_recall": 0,
"validation__arg_class_f1": 0,
"validation__args_multiple": 0,
"validation_loss": 0.3997326233545447,
"best_validation__coref_precision": 0.0,
"best_validation__coref_recall": 0.0,
"best_validation__coref_f1": 0.0,
"best_validation__coref_mention_recall": 0.0,
"best_validation__ner_precision": 0.0,
"best_validation__ner_recall": 0.0,
"best_validation__ner_f1": 0.0,
"best_validation_rel_precision": 0.00013013713035796913,
"best_validation_rel_recall": 0.037037037037037035,
"best_validation_rel_f1": 0.00025936293651240686,
"best_validation_rel_span_recall": 0.04526748971193416,
"best_validation__trig_id_precision": 0,
"best_validation__trig_id_recall": 0,
"best_validation__trig_id_f1": 0,
"best_validation__trig_class_precision": 0,
"best_validation__trig_class_recall": 0,
"best_validation__trig_class_f1": 0,
"best_validation__arg_id_precision": 0,
"best_validation__arg_id_recall": 0,
"best_validation__arg_id_f1": 0,
"best_validation__arg_class_precision": 0,
"best_validation__arg_class_recall": 0,
"best_validation__arg_class_f1": 0,
"best_validation__args_multiple": 0,
"best_validation_loss": 0.15924045607033888
}