nlp_model.rst

Model NLP

The config file for NLP models contain three main sections:

• trainer: contains the configs for PTL training. For more information, refer to :doc:`../core/core` and PTL Trainer class API <https://pytorch-lightning.readthedocs.io/en/stable/common/trainer.html#trainer-class-api>.
• exp_manager: the configs of the experiment manager. For more information, refer to :doc:`../core/core`.
• model: contains the configs of the datasets, model architecture, tokenizer, optimizer, scheduler, etc.

The following sub-sections of the model section are shared among most of the NLP models.

• tokenizer: specifies the tokenizer
• language_model: specifies the underlying model to be used as the encoder
• optim: the configs of the optimizer and scheduler :doc:`../core/core`

The tokenizer and language_model sections have the following parameters:

Parameter	Data Type	Description
model.tokenizer.tokenizer_name	string	Tokenizer name will be filled automatically based on model.language_model.pretrained_model_name.
model.tokenizer.vocab_file	string	Path to tokenizer vocabulary.
model.tokenizer.tokenizer_model	string	Path to tokenizer model (only for sentencepiece tokenizer).
model.language_model.pretrained_model_name	string	Pre-trained language model name, for example: bert-base-cased or bert-base-uncased.
model.language_model.lm_checkpoint	string	Path to the pre-trained language model checkpoint.
model.language_model.config_file	string	Path to the pre-trained language model config file.
model.language_model.config	dictionary	Config of the pre-trained language model.

The parameter model.language_model.pretrained_model_name can be one of the following:

• megatron-bert-345m-uncased, megatron-bert-345m-cased, biomegatron-bert-345m-uncased, biomegatron-bert-345m-cased, bert-base-uncased, bert-large-uncased, bert-base-cased, bert-large-cased
• distilbert-base-uncased, distilbert-base-cased
• roberta-base, roberta-large, distilroberta-base
• albert-base-v1, albert-large-v1, albert-xlarge-v1, albert-xxlarge-v1, albert-base-v2, albert-large-v2, albert-xlarge-v2, albert-xxlarge-v2