Save weights at any time between integral epochs

[Yochengliu]

In caffe, one can save the model weights at any time, for example, in iterations of 32768. Just like pressing 'Ctrl + C', then the training process stops and the model weights are saved automatically.

But in mxnet, I can't find a way like in caffe to save weights at any time, it seems that mxnet can only save weights in integral epochs, such as epoch 1, 2, 3 ...

I feel sorry, because for large-scale datasets, it takes several days for training for one epoch. So, if something wrong, then a lot of time is wasted.

Any solutions for this ? Please in detail.
Thanks.

[solin319]

1. Try to add follow code after batch_end_callback in 'base_module.py'.

arg_params, aux_params = self.get_params()
if epoch_end_callback is not None:
                for callback in _as_list(epoch_end_callback):
                    callback(nbatch, self.symbol, arg_params, aux_params)

2. When define 'mx.callback.do_checkpoint', you can pass 'period' to control number of iterations to save checkpoints.

[Yochengliu]

@solin319 Thank you, I will have a try, and do you know some specific saving ways that just press 'Ctrl + C', then the training process stops and the model weights are saved automatically ?

[szha]

@apache/mxnet-committers: This issue has been inactive for the past 90 days. It has no label and needs triage.

For general "how-to" questions, our user forum (and Chinese version) is a good place to get help.

[lanking520]

Hi @Yochengliu , I would like to follow up this topic and find someone who can possibly help you. Have you found any ways to meet your needs? @nswamy can you label this as 'feature request'?