Predictions on MathQA using pre-trained model

[kbogas]

Hello! Thanks for making available your work and models, alongside steps to reproduce the results.

I am trying to generate the predictions on the MathQA dataset, using the trained monolingual_en model that you provide us with in the google drive link.

Essentially, I am using the test_model found in run_ft with minor modifications for paths and for keeping the results from compute_prefix_tree_result.

However, the results I get are greatly lower than the ones report (accuracy at around 0.03%), so obviously I am not running something correctly.

What I am doing is similar to running something along the lines:

python run_ft.py
--output_dir model_monolingual_en
--bert_pretrain_path ./pretrained_models_vocab/bert-base-uncased
--data_dir data
--train_file MathQA_bert_token_train.json
--test_file MathQA_bert_token_test.json
--hidden_size 768
--only_test

(obviously I am loading/handling correctly the paths in the changed test_model for my purpose).

Is there something wrong with this approach? Should I be using a different model altogether?

Thank you for your time.

[ChaoYan]

Hello, you can try adjusting the learning rate to 5e-5, and then observe the results.

[kbogas]

@ChaoYan Thank you for the response.

I am not trying to train or fine-tune the model, I am just trying to use the trained model (after stage 2 as mentioned), provided by the authors here, in order to get the predictions on the MathQA test-set.

I was hoping the model I am using from the link is the final version of the one named "Monolingual Setting BERT-TD w CL" that achieves 76.3% accuracy in MathQA (with no need for further fine-tuning).

If am using a wrong model or process for inference, I would be thankful If you could provide me with instructions on this. Or if there are available somewhere the predictions of the model on the test-set of the MathQA directly, I would be glad if you could share them.

Once again, thank you for making your code public!

[zwx980624]

For loading a model, you need to add " --model_reload_path model_monolingual_en " like
python run_ft.py
--output_dir output/mwp-ft-monolingual-en
--bert_pretrain_path pretrained_models/bert-base-uncased
--data_dir data
--train_file MathQA_bert_token_train.json
--finetune_from_trainset MathQA_bert_token_train.json
--dev_file MathQA_bert_token_val.json
--test_file MathQA_bert_token_test.json
--only_test
--model_reload_path model_monolingual_en
And remember to change the vocab.txt in pretrained_models/bert-base-uncased

[ICanFlyGFC]

Hello! Thanks for making your work and models available, alongside steps to reproduce the results.

I follow your guideline, but I make mistakes. It's like the trained model is mismatched with the neural network structure. What should I do to run the code successfully? Looking forward to your reply and thanks! @zwx980624

python run_ft.py
--output_dir output/mwp-ft-monolingual-en
--bert_pretrain_path pretrained_models/bert-base-uncased
--data_dir data --train_file MathQA_bert_token_train.json
--finetune_from_trainset MathQA_bert_token_train.json
--dev_file MathQA_bert_token_val.json
--test_file MathQA_bert_token_test.json
--only_test
--model_reload_path mwp_trained_model/model_monolingual_en
2022-07-19 22:18:57,573 - main - INFO - output_dir: output/mwp-ft-monolingual-en
2022-07-19 22:18:57,574 - main - INFO - bert_pretrain_path: pretrained_models/bert-base-uncased
2022-07-19 22:18:57,574 - main - INFO - train_file: MathQA_bert_token_train.json
2022-07-19 22:18:57,574 - main - INFO - model_reload_path: mwp_trained_model/model_monolingual_en
2022-07-19 22:18:57,574 - main - INFO - finetune_from_trainset: MathQA_bert_token_train.json
2022-07-19 22:18:57,574 - main - INFO - data_dir: data
2022-07-19 22:18:57,574 - main - INFO - dev_file: MathQA_bert_token_val.json
2022-07-19 22:18:57,574 - main - INFO - test_file: MathQA_bert_token_test.json
2022-07-19 22:18:57,574 - main - INFO - schedule: linear
2022-07-19 22:18:57,574 - main - INFO - batch_size: 16
2022-07-19 22:18:57,574 - main - INFO - learning_rate: 5e-05
2022-07-19 22:18:57,574 - main - INFO - n_epochs: 50
2022-07-19 22:18:57,574 - main - INFO - max_grad_norm: 3.0
2022-07-19 22:18:57,574 - main - INFO - warmup_steps: 0
2022-07-19 22:18:57,574 - main - INFO - n_save_ckpt: 1
2022-07-19 22:18:57,574 - main - INFO - n_val: 5
2022-07-19 22:18:57,574 - main - INFO - logging_steps: 100
2022-07-19 22:18:57,574 - main - INFO - embedding_size: 128
2022-07-19 22:18:57,574 - main - INFO - hidden_size: 512
2022-07-19 22:18:57,574 - main - INFO - beam_size: 5
2022-07-19 22:18:57,574 - main - INFO - dropout: 0.5
2022-07-19 22:18:57,574 - main - INFO - seed: 42
2022-07-19 22:18:57,574 - main - INFO - only_test: True
bert_tokenizer pretrained_models/bert-base-uncased
Indexing words...
keep_words 4796 / 6804 = 0.7049
Indexed 4799 words in input language, 52 words in output
Number of training data 23703
Number of testind data 2410
Number of testind data 3540
bert_model: pretrained_models/bert-base-uncased
Some weights of the model checkpoint at pretrained_models/bert-base-uncased were not used when initializing BertModel: ['cls.predictions.decoder.weight', 'cls.predictions.transform.dense.weight', 'cls.seq_relationship.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.bias', 'cls.predictions.transform.dense.bias', 'cls.seq_relationship.weight', 'cls.predictions.transform.LayerNorm.bias']

• This IS expected if you are initializing BertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
• This IS NOT expected if you are initializing BertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
  Traceback (most recent call last):
  File "D:/WorkSpace/mwp-cl-main/run_ft.py", line 349, in
  encoder, predict, generate, merge = initial_model(output_lang, embedding_size, hidden_size, args, copy_nums, generate_nums)
  File "D:/WorkSpace/mwp-cl-main/run_ft.py", line 54, in initial_model
  encoder.load_state_dict(torch.load(os.path.join(args.model_reload_path, "encoder.ckpt")))
  File "D:\Anaconda\envs\mvp-cl\lib\site-packages\torch\nn\modules\module.py", line 1483, in load_state_dict
  self.class.name, "\n\t".join(error_msgs)))
  RuntimeError: Error(s) in loading state_dict for EncoderBert:
  size mismatch for linear.weight: copying a param with shape torch.Size([768, 768]) from checkpoint, the shape in current model is torch.Size([512, 768]).
  size mismatch for linear.bias: copying a param with shape torch.Size([768]) from checkpoint, the shape in current model is torch.Size([512]).

Process finished with exit code 1

[dilaraIsikliIntellico]

Hi @ICanFlyGFC ! I have received the same problem as you. could you solve it later? If you could share your experience with me, it would be so helpful, thanks :)