[feature request] Model averaging

[vince62s]

In order to do model averaging, we need to keep several checkpoints.
2 approaches:
Time-based: this is the TF approach which keeps 10 minutes interval ckpt files, up to N (20 by default) files
Step-based: same as in Lua-onmt with save_every option which saves a chekcpoint every X iteration.

Step is more appropriate since system's speed could vary a lot.
But it's also good to have a "keep last N" flag to minimize disk usage.

then the external tool can average weights and spit out the averaged model.
should guve at least 1 bleu improvement.

[vince62s]

Thanks @pltrdy
The last thing we need is 2 extra options at training
-keep_ckpt N
-save_ckpt_every M

N means we keep the last N checkpoint (eg: 5 10 20) good for averaging.
M is a number of iterations. We save a checkpoint each M iteration.

@srush @helson73 tell me if you're ok with this.

[helson73]

That’s great!