added model saving, loading and checkpointing support to PyTorch

[dmudiger]

Adds model saving/loading support for the PyTorch implementation -
--save-model="model file" (e.g: model.pt)
--load-model="model file" (e.g: model.pt)

Also, the saved model can be used with --inference-only to run only testing on a previously trained (and saved) model.

[berryweinst]

I ran this command line after your addition:
CUDA_VISIBLE_DEVICES=0 python dlrm_s_pytorch.py --arch-sparse-feature-size=16 --arch-mlp-bot="13-512-256-64-16" --arch-mlp-top="512-256-1" --data-generation=dataset --data-set=kaggle --processed-data-file=./kaggle_data/kaggleAdDisplayChallenge_processed.npz --loss-function=bce --round-targets=True --learning-rate=0.1 --mini-batch-size=128 --print-freq=1024 --print-time --raw-data-file /media/drive/Datasets/criteo/train.txt --use-gpu --save-model ./model/checkpoint.pt

But nothing was saved. I removed the condition in the main script as a WA for now to see if it works, but the main problem is that this stage of reading the batched takes all day long, and so if the model wasn't saved (or any other issue) I need to start it all over again. Is there anything to do with it?

Converted to tensors...done!
Sparse features = 26, Dense features = 13
Training data
Total number of batches 263115
Reading in batch 14345/263115

[berryweinst]

After it finished to count the data it crashed.

Reading in batch: 263115 / 263115

Testing data
Total number of batches 21926
Reading in batch: 21926 / 21926

time/loss/accuracy (if enabled):
Saving model to ./model/checkpoint.pt
Traceback (most recent call last):
File "dlrm_s_pytorch.py", line 855, in
"train_acc": gA,
NameError: name 'gA' is not defined

[dmudiger]

Hi, can you try withy the latest version of the code since I'm not able to see the same issue at my end.

Also, the model is saved only when the testing is done. So you would need to add --test-freq argument with a number of iterations interval for doing the testing. It would be best to set this to be sufficiently high since it this additional overhead of having to do evaluate on the whole test data-set.

[berryweinst]

I guess what was missing the --test-freq. Now it seem to work.

[dmudiger]

Happy to hear!

Also, for quick tests you can use the —num-batches argument to quickly test by running with only a limited number of batches and once it works you can run with the full data-set

[berryweinst]

Thanks a lot.
As for the data preparing. This reading stage (Reading in batch 14345/263115) can take all day long before the training starts. What exactly is taking so long? Is it the read and process each record in Criteo? Is there a way you can do this in a threaded way inside the batch pre-fetch with even a few workers involved?