the bad result

[cccccs]

hi,I run this code fluently，But the result is bad，I don't know what’s wrong with this. could you please give me some suggestions？thanks。

python -u run_scorer_test.py ./models/pairwise_scorers/test_events_average_0.8_model_3_corpus_level.conll events Processing file: ./models/pairwise_scorers/test_events_average_0.8_model_3_corpus_level.conll 0.65 1.0 0.787878787878788 0.8205128205128205 0.8205128205128205 0.8205128205128205 0.5474218379209004 0.6790832621385336 0.6061858323184838 0.15918234423790356 0.7040757533599581 0.2596591430831051 0.4649510356293238 0.6254223508155559 0.5333783332066294 {'mentions_recall': 0.65, 'mentions_precision': 1.0, 'mentions_f1': 0.787878787878788, 'muc_recall': 0.8205128205128205, 'muc_precision': 0.8205128205128205, 'muc_f1': 0.8205128205128205, 'bcub_recall': 0.5474218379209004, 'bcub_precision': 0.6790832621385336, 'bcub_f1': 0.6061858323184838, 'ceafe_recall': 0.15918234423790356, 'ceafe_precision': 0.7040757533599581, 'ceafe_f1': 0.2596591430831051, 'lea_recall': 0.4649510356293238, 'lea_precision': 0.6254223508155559, 'lea_f1': 0.5333783332066294, 'conll': 56.211926530480305}

[cccccs]

hi,I run this code fluently，But the result is bad，I don't know what’s wrong with this. could you please give me some suggestions？thanks。

`python -u run_scorer_test.py ./models/pairwise_scorers/test_events_average_0.8_model_3_corpus_level.conll events Processing file: ./models/pairwise_scorers/test_events_average_0.8_model_3_corpus_level.conll 0.65 1.0 0.787878787878788 0.8205128205128205 0.8205128205128205 0.8205128205128205 0.5474218379209004 0.6790832621385336 0.6061858323184838 0.15918234423790356 0.7040757533599581 0.2596591430831051 0.4649510356293238 0.6254223508155559 0.5333783332066294 {'mentions_recall': 0.65, 'mentions_precision': 1.0, 'mentions_f1': 0.787878787878788, 'muc_recall': 0.8205128205128205, 'muc_precision': 0.8205128205128205, 'muc_f1': 0.8205128205128205, 'bcub_recall': 0.5474218379209004, 'bcub_precision': 0.6790832621385336, 'bcub_f1': 0.6061858323184838, 'ceafe_recall': 0.15918234423790356, 'ceafe_precision': 0.7040757533599581, 'ceafe_f1': 0.2596591430831051, 'lea_recall': 0.4649510356293238, 'lea_precision': 0.6254223508155559, 'lea_f1': 0.5333783332066294, 'conll': 56.211926530480305

about this question，do I also need to play with hyperparameter？
my hyperparameter is：
{
"gpu_num" : [1],
"bert_model": "roberta-large",
"bert_hidden_size": 1024,
"hidden_layer": 1024,
"dropout": 0.3,
"with_mention_width": true,
"with_head_attention": true,
"embedding_dimension": 20,

"max_mention_span": 10,
"use_gold_mentions": true,
"mention_type": "events",
"top_k": 0.25,
"training_method": "continue",
"subtopic": true,
"use_predicted_topics": false,
"segment": true,

"random_seed": 0,
"epochs":10,
"batch_size": 128,
"learning_rate": 1e-4,
"weight_decay": 0,
"loss": "bce",
"optimizer": "adam",
"adam_epsilon": 1e-8,
"segment_window": 512,
"neg_samp": true,
"exact": false,

"log_path": "logs/pairwise_scorer/",
"data_folder": "data/ecb/mentions",
"span_repr_path": "models/span_scorers/events_span_repr_0",
"span_scorer_path": "models/span_scorers/events_span_scorer_0",
"model_path": "models/pairwise_scorers"
}
could you give me some suggestions？ thanks.

[cccccs]

hi,I run this code fluently，But the result is bad，I don't know what’s wrong with this. could you please give me some suggestions？thanks。

python -u run_scorer_test.py ./models/pairwise_scorers/test_events_average_0.8_model_3_corpus_level.conll events Processing file: ./models/pairwise_scorers/test_events_average_0.8_model_3_corpus_level.conll 0.65 1.0 0.787878787878788 0.8205128205128205 0.8205128205128205 0.8205128205128205 0.5474218379209004 0.6790832621385336 0.6061858323184838 0.15918234423790356 0.7040757533599581 0.2596591430831051 0.4649510356293238 0.6254223508155559 0.5333783332066294 {'mentions_recall': 0.65, 'mentions_precision': 1.0, 'mentions_f1': 0.787878787878788, 'muc_recall': 0.8205128205128205, 'muc_precision': 0.8205128205128205, 'muc_f1': 0.8205128205128205, 'bcub_recall': 0.5474218379209004, 'bcub_precision': 0.6790832621385336, 'bcub_f1': 0.6061858323184838, 'ceafe_recall': 0.15918234423790356, 'ceafe_precision': 0.7040757533599581, 'ceafe_f1': 0.2596591430831051, 'lea_recall': 0.4649510356293238, 'lea_precision': 0.6254223508155559, 'lea_f1': 0.5333783332066294, 'conll': 56.211926530480305}

The result seems to be predicted model，how can I run the gold model？I have set use_gold_mentions true. but this seems to be the result of predicted model. thanks.

[cccccs]

hi,I run this code fluently，But the result is bad，I don't know what’s wrong with this. could you please give me some suggestions？thanks。
python -u run_scorer_test.py ./models/pairwise_scorers/test_events_average_0.8_model_3_corpus_level.conll events Processing file: ./models/pairwise_scorers/test_events_average_0.8_model_3_corpus_level.conll 0.65 1.0 0.787878787878788 0.8205128205128205 0.8205128205128205 0.8205128205128205 0.5474218379209004 0.6790832621385336 0.6061858323184838 0.15918234423790356 0.7040757533599581 0.2596591430831051 0.4649510356293238 0.6254223508155559 0.5333783332066294 {'mentions_recall': 0.65, 'mentions_precision': 1.0, 'mentions_f1': 0.787878787878788, 'muc_recall': 0.8205128205128205, 'muc_precision': 0.8205128205128205, 'muc_f1': 0.8205128205128205, 'bcub_recall': 0.5474218379209004, 'bcub_precision': 0.6790832621385336, 'bcub_f1': 0.6061858323184838, 'ceafe_recall': 0.15918234423790356, 'ceafe_precision': 0.7040757533599581, 'ceafe_f1': 0.2596591430831051, 'lea_recall': 0.4649510356293238, 'lea_precision': 0.6254223508155559, 'lea_f1': 0.5333783332066294, 'conll': 56.211926530480305}

The result seems to be predicted model，how can I run the gold model？I have set use_gold_mentions true. but this seems to be the result of predicted model. thanks.

could you tell me the config's settings of the best gold model? such as the settings of use_gold_mentions、subtopic、use_predicted_topics、topic_level、keep_singletons. when I set use_gold_mentions true,subtopic true,use_predicted_topics false,topic_level true,keep_singletons false, the result of conll f1 is just 71.How can I get the result of 81? thanks very mach.

[cccccs]

Oh, I have know the reason why the result is bad, the get_ecb_data.py get the dataset without singletons, but the best result is with the singletons, so the save_gold_conll_files in get_ecb_data.py should be ：

def save_gold_conll_files(documents, mentions, clusters, dir_path, doc_name):
    #non_singletons = {cluster: ms for cluster, ms in clusters.items() if len(ms) > 1}
    singletons = {cluster:ms for cluster,ms in clusters.items()}
    doc_ids = [m['doc_id'] for m in mentions]
    starts = [min(m['tokens_ids']) for m in mentions]
    ends = [max(m['tokens_ids']) for m in mentions]
    write_output_file(documents, singletons, doc_ids, starts, ends, dir_path, doc_name)

and the subtopic = true，topic_level = true，keep_singletons = true.

[LinfanLiu01]

Hello, have you reached the score of 81 in the paper? My running result is always around 79