-
Notifications
You must be signed in to change notification settings - Fork 48
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Error occurred while using sentence-level Predictor-Estimator to predict #23
Comments
Hello Zachary, I was not able to reproduce your bug. Can you verify that you are using the newest version of the repository? If you already do or that does not solve your problem, can you provide the trained model for us to analyze what is the problem? Best |
Thank you for your reply. https://www.dropbox.com/s/ce9akwcvhs4tcbo/best_model.torch?dl=0 My config file for training train_estimator_yaml.txt It's worth noting that I trained 2019-04-24 04:03:26.789 [root setup:380] This is run ID: 62e6dc469e3a4971bbce19bc119487c5 Thanks a lot! |
Hi @Zachary-YL we will look into what is happening on the predict pipeline. Meanwhile, the error that you're getting when training on a GPU just means that OpenKiwi is trying to allocate more memory than what is available on your GPU. This happens when the combination of Batch_size and N of tokens on a sentence is too large. You can easily train using the GPU if you do one of two things (or both):
|
Hello Zachary, Sorry for the long delay in response, our team was busy with the WMT shared task. I have run your predict-yaml with the model you provided (changing source and target to a toy file) and it worked fine without error. |
I am closing this as it seems to be solved. |
After successfully training the sentence-level
Predictor
andEstimator
model,an error occurred while using theEstimator
model to predict the sentence-level data.The command is:
kiwi predict --config experiments_sl/predict_estimator.yaml
And the error is:
2019-04-22 07:19:37.521 [kiwi.lib.predict setup:159] {'batch_size': 64,
'config': 'experiments_sl/predict_estimator.yaml',
'debug': False,
'experiment_name': 'EN-ZH Pretrain Predictor',
'gpu_id': None,
'load_data': None,
'load_model': 'runs/0/464dc10bfc174ac79ca082eae0dea352/best_model.torch',
'load_vocab': None,
'log_interval': 100,
'mlflow_always_log_artifacts': False,
'mlflow_tracking_uri': 'mlruns/',
'model': 'estimator',
'output_dir': 'predictions/predest/ccmt/en_zh',
'quiet': False,
'run_uuid': None,
'save_config': None,
'save_data': None,
'seed': 42}
2019-04-22 07:19:37.521 [kiwi.lib.predict setup:160] Local output directory is: predictions/predest/ccmt/en_zh
2019-04-22 07:19:37.521 [kiwi.lib.predict run:100] Predict with the PredEst (Predictor-Estimator) model
Traceback (most recent call last):
File "/home2/zyl/anaconda3/envs/openkiwi/bin/kiwi", line 10, in
sys.exit(main())
File "/home2/zyl/anaconda3/envs/openkiwi/lib/python3.6/site-packages/kiwi/main.py", line 22, in main
return kiwi.cli.main.cli()
File "/home2/zyl/anaconda3/envs/openkiwi/lib/python3.6/site-packages/kiwi/cli/main.py", line 73, in cli
predict.main(extra_args)
File "/home2/zyl/anaconda3/envs/openkiwi/lib/python3.6/site-packages/kiwi/cli/pipelines/predict.py", line 56, in main
predict.predict_from_options(options)
File "/home2/zyl/anaconda3/envs/openkiwi/lib/python3.6/site-packages/kiwi/lib/predict.py", line 54, in predict_from_options
run(options.model_api, output_dir, options.pipeline, options.model)
File "/home2/zyl/anaconda3/envs/openkiwi/lib/python3.6/site-packages/kiwi/lib/predict.py", line 113, in run
model = Model.create_from_file(pipeline_opts.load_model)
File "/home2/zyl/anaconda3/envs/openkiwi/lib/python3.6/site-packages/kiwi/models/model.py", line 214, in create_from_file
model = Model.subclasses[model_name].from_dict(model_dict)
File "/home2/zyl/anaconda3/envs/openkiwi/lib/python3.6/site-packages/kiwi/models/model.py", line 235, in from_dict
model.load_state_dict(class_dict[const.STATE_DICT])
File "/home2/zyl/anaconda3/envs/openkiwi/lib/python3.6/site-packages/torch/nn/modules/module.py", line 769, in load_state_dict
self.class.name, "\n\t".join(error_msgs)))
RuntimeError: Error(s) in loading state_dict for Estimator:
Unexpected key(s) in state_dict: "predictor_tgt.W2", "predictor_tgt.V", "predictor_tgt.C", "predictor_tgt.S", "predictor_tgt.attention.scorer.layers.0.0.weight", "predictor_tgt.attention.scorer.layers.0.0.bias", "predictor_tgt.attention.scorer.layers.1.0.weight", "predictor_tgt.attention.scorer.layers.1.0.bias", "predictor_tgt.embedding_source.weight", "predictor_tgt.embedding_target.weight", "predictor_tgt.lstm_source.weight_ih_l0", "predictor_tgt.lstm_source.weight_hh_l0", "predictor_tgt.lstm_source.bias_ih_l0", "predictor_tgt.lstm_source.bias_hh_l0", "predictor_tgt.lstm_source.weight_ih_l0_reverse", "predictor_tgt.lstm_source.weight_hh_l0_reverse", "predictor_tgt.lstm_source.bias_ih_l0_reverse", "predictor_tgt.lstm_source.bias_hh_l0_reverse", "predictor_tgt.lstm_source.weight_ih_l1", "predictor_tgt.lstm_source.weight_hh_l1", "predictor_tgt.lstm_source.bias_ih_l1", "predictor_tgt.lstm_source.bias_hh_l1", "predictor_tgt.lstm_source.weight_ih_l1_reverse", "predictor_tgt.lstm_source.weight_hh_l1_reverse", "predictor_tgt.lstm_source.bias_ih_l1_reverse", "predictor_tgt.lstm_source.bias_hh_l1_reverse", "predictor_tgt.forward_target.weight_ih_l0", "predictor_tgt.forward_target.weight_hh_l0", "predictor_tgt.forward_target.bias_ih_l0", "predictor_tgt.forward_target.bias_hh_l0", "predictor_tgt.forward_target.weight_ih_l1", "predictor_tgt.forward_target.weight_hh_l1", "predictor_tgt.forward_target.bias_ih_l1", "predictor_tgt.forward_target.bias_hh_l1", "predictor_tgt.backward_target.weight_ih_l0", "predictor_tgt.backward_target.weight_hh_l0", "predictor_tgt.backward_target.bias_ih_l0", "predictor_tgt.backward_target.bias_hh_l0", "predictor_tgt.backward_target.weight_ih_l1", "predictor_tgt.backward_target.weight_hh_l1", "predictor_tgt.backward_target.bias_ih_l1", "predictor_tgt.backward_target.bias_hh_l1", "predictor_tgt.W1.weight".
Could you give some advice for solving this error? Thanks a lot!
The text was updated successfully, but these errors were encountered: