-
Notifications
You must be signed in to change notification settings - Fork 73
FAQs
A: Neural engines can handle multiple language pairs. To deploy one multilingual neural engine, you can follow this guide.
A: It's very easy. First, you can create your engines one by one with the mmt create command.
You must choose different names for the engines: if you don't, all engines will use the name "default" so each new engine will overwrite the previous one. You can choose engine names with the -e option in mmt create.
Example:
$ ./mmt create en it examples/data/train -e engine1
$ ./mmt create it en examples/data/train -e engine2
After that, you can start your engines one by one with the ./mmt start command.
Each engine must use different logic ports of your machine, so make sure to pass correct values for all the necessary options:
-p (for API port; default value: 8045),
--cluster-port (default value: 5016),
--datastream-port (default value: 9092)
--db-port (default value: 9042)
Example:
# this way engine1 will use the default values for all ports...
$ ./mmt start -e engine1
# ...and engine2 will use different values
$ ./mmt start -e engine2 -p 8046 --cluster-port 5017 --datastream-port 9093 --db-port 9043A: MMT extracts them randomly on the fly from the whole training corpus. By default:
- 1200 sentence pairs are extracted as develop/validation set. They are stored in
ModernMT/engines/<your_engine>/data/dev - 1200 sentence pairs are extracted as test set. They are stored in
ModernMT/engines/<your_engine>/data/test
During preprocessing the dev/validation set is used as Validation set in Neural Engine training (if not specified), or for development set for Phrase-Based tuning (if not specified).
The test set is used for ./mmt evaluate if no --path option is specified.
Q: Is there any difference in the data needed to train the new neural engines, in comparison with the old phrase-based ones?
A: Basically, both neural engines and phrase-based engines need parallel corpora for their training.
However, while phrase-based engines also needed monolingual resources to build their language models, neural engines do not need any monolingual data. If in your training corpora folder contains monolingual files, a neural engine will thus just ignore them.
Q: When you import a TMX file to an engine, how does it process this file? Does it build on the current model and start training from there?
A: Our Neural Engine it's basically made of two main components: the neural model and the translation memory manager.
The neural model is built during the training and after that moment, it won't change anymore: it is the baseline static model.
However, when you import a TMX file, it is added only to the translation memory manager and used for the adaptation. This way, we can use the most suitable the sentence pairs in each TM to adapt on the fly the neural (baseline-static) model, in order to improve the translation quality.
A: Yes, you can perform a warm-up before starting your engine.
After your engine has been created, please run this command from your MMT root folder:
find engines/<your_engine_name>/models/ -type f -exec dd if={} of=/dev/null bs=1024000 \;This will fill your OS RAM buffer with all the models for your engine.
This way your models will be accessed faster, boosting your performances since the very beginning of the engine runtime.
Q: Sometimes I have a problem with specific files when using them to train an engine, or when adding them to an already trained engine. Why does this happen?
A: That may depend on the file structure or encoding. If the file is a TMX, please make sure that it follows the TMX structure rules. It may also depend on the file encoding; by design, ModernMT only supports UTF-8 encoded files. If your files are not already encoded with UTF-8, please convert them to UTF-8 before using them in MMT.
A: the Matecat score works in the same direction of the BLEU score. It is a similarity score between MT output and reference, thus the higher it is the less one needs to post-edit.
A: in its cleaning phase, that is performed right before pre-processing, MMT filters out low quality segments from the input corpora because they may affect the overall quality of the resulting engine. The error you are experiencing is caused by the fact that, after cleaning, in your input corpora there were no segments left for the language direction of your engine.
Of course, the first suggestion is to employ a higher quality corpus; if you don't have one, you can probably find some good corpora here: http://opus.nlpl.eu/
Another possibility is to skip the cleaning phase. MMT allows to manually specify which steps to run during ./mmt create, so you can just specify all steps but the cleaning one. Please note that skipping the cleaning phase nay heavily affect resulting engine quality, though.
- In phrase-based engines, the creation steps are clean_tms, preprocess, train_context, train_aligner, train_tm, train_lm. So, in order to execute the ./mmt create command skipping the
clean_tmsstep, just open a bash in your MMT home folder and type:
./mmt create <your_src_lang> <your_trg_lang> /path/to/your_corpora -s \
preprocess \
train_context \
train_aligner \
train_tm \
train_lm- In neural engines, the creation steps are clean_tms, preprocess, train_context, train_aligner, build_memory, prepare_training_data, train_decoder. So, in order to execute the ./mmt create command skipping the
clean_tmsstep, just open a bash in your MMT home folder and type:
./mmt create <your_src_lang> <your_trg_lang> /path/to/your_corpora --neural -s \
preprocess \
train_context \
train_aligner \
build_memory \
prepare_training_data \
train_decoder