Extract the word-counts to each Trainer

[n1t0]

Current state

Training a Model starts by computing the word counts from the training corpus, in the Tokenizer. We later provide these word counts to the relevant Trainer in order to start the training. This actually has several limitations:

• Computing the word counts is not always the best starting point to train a Model
• This prevents streaming the corpus directly in the Trainer, while forcing us to build a first representation in memory. This is limiting for big datasets. Sometimes Trainers can directly build a better representation, effectively reducing the memory footprint.

Goal

Change the Trainer API to:

• Feed it with &str directly
• Leave it the responsibility to build its own representation
• train should just take the Model to train.

[pietrolesci]

HI @n1t0,

Possibly related to this: is there a way from the users' side to get the word counts after training a tokenizer?

I preferred asking this here instead of opening an issue directly.
Context: I am on '0.10.0rc1' using the WordLevel tokenizer