Add pause/resume capability for the models trainer

[iustinam90]

Expected Result

Save the current trainer state (weights, costs, etc) to files on disk to be able to stop and resume training at a later time.

Actual Result

Currently, when the trainer is interrupted, no state is saved so that when you rerun the trainer it will start from scratch.

[tiberiu44]

Actually, we do have checkpoints saved after each epoch, but there is no resume capability. This should not be too hard. Each network is able to load a pretrained model. We only have to restructure the entry training method to check if a model was provided via command line and load it after it initializes the network

[ruxandraburtica]

We need to do the following:

• Keep track of model metrics, and save them to disk
• Load model from the path where it was saved
• Load model metrics from disk

embeddings = WordEmbeddings()
embeddings.read_from_file(params.embeddings, encodings.word_list)

# Load parser from file, instead of using BDRNNParser.
path = params.output_base + '.last'
print('Loading model from path: {}'.format(path))
parser.load(path)

trainer = ParserTrainer(parser, encodings, params.itters, trainset, devset, testset)
trainer.start_training(params.output_base, params.batch_size)

[tiberiu44]

This is a rare use-case. It's not speech/voice processing. Training is a lot faster and it does not require resume capabilities.