[air] Add horovod trainer. #23437
[air] Add horovod trainer. #23437
Conversation
| scaling_config=scaling_config, | ||
| ) | ||
| trainer.fit() | ||
|
|
There was a problem hiding this comment.
Could we also e2e test with TorchPredictor as well, like what we have in test_torch_trainer?
There was a problem hiding this comment.
I find it very heard to use the predictor interface for this image classification problem.
For the sake of verifying the training process, I just use the native pytorch DataLoader and Tensor stuff (and not the predictor).
I may need to just put a linear training if we want to cover the predictor part in an e2e fashion for horovod.
Although in terms of test coverage, I think the corresponding test_pytorch/tensorflow_trainer got it covered already.
I find it very heard to use the predictor interface for this image classification problem. For the sake of verifying the model, I just use the native pytorch DataLoader and Tensor stuff.
| num_epochs = config.get("num_epochs", 10) | ||
| log_interval = config.get("log_interval", 10) | ||
| use_cuda = config.get("use_cuda", False) | ||
| save_model_as_dict = config.get("save_model_as_dict", False) |
There was a problem hiding this comment.
Seems like this is always going to be False?
There was a problem hiding this comment.
hmmm, I have a test, where save_model_as_dict is True so that we can test that path as well. So it should be taking effect.
|
|
||
| .. code-block:: python | ||
|
|
||
| class Net(nn.Module): |
There was a problem hiding this comment.
can we use a simpler example for the docstring 🙂
I think we can just copy the one for TorchTrainer, except add in the hvd.init(), hvd.DistributedOptimizer, etc. lines.
There was a problem hiding this comment.
Yes, good point.
I updated with a simple linear example instead.
Why are these changes needed?
Related issue number
Checks
scripts/format.shto lint the changes in this PR.