Compatibility with PTL 1.6

[krfricke]

Todos:

• Check if we need to_state_stream / load_state_stream P(0)
• Check multi node (P0)
• Check multi GPU/multi node (P0)
• Fix / change tests (P0)
• Check that recent PRs are included, e.g. ray_ddp: support logged_metrics as part of remote worker return value #156 P(0.5-1)
• Check Ray client (P1)
• Check fractional GPUs (P2)
• DDP sharded (P2)

[amogkam]

DDP Sharded and Horovod also need to be P0. We have to make sure the entire library works with 1.6.

[amogkam]

DDP Sharded and Horovod also need to be P0. We have to make sure the entire library works with 1.6.

[amogkam]

Could we also merge in this PR to rename everything to strategy following the 1.6 convention. #129

[JiahaoYao]

Error:

• enable gpu training

[JiahaoYao]

- [ ] multi-node gpu case, it hangs up forever.

[JiahaoYao]

to_state_stream / load_state_stream seems to be useful because
the weights will not be changed

from this test:

ray_lightning/ray_lightning/tests/utils.py

Lines 241 to 245 in 6aed848

	post_train_values = torch.tensor(
	[torch.sum(torch.abs(x)) for x in model.parameters()])
	assert trainer.state.finished, f"Trainer failed with {trainer.state}"
	# Check that the model is actually changed post-training.
	assert torch.norm(initial_values - post_train_values) > 0.1

[JiahaoYao]

for the Q1:

Check if we need to_state_stream / load_state_stream P(0)

Yes, we need this. On the remote task, the weights is not going to dump / load from the ckpt from the hard-disk. And to_state_stream / load_state_stream provides a elegant way to fetch and pass the weights to driver.

[JiahaoYao]

passed all the test except multi-gpu on one node for this version (https://github.com/JiahaoYao/ray_lightning/tree/3df599a8bb1ac917bf6352b8baef63ad64f21595)

[JiahaoYao]

the test files used are https://github.com/JiahaoYao/ray_lightning/blob/3df599a8bb1ac917bf6352b8baef63ad64f21595/ray_lightning/tests/test_ddp_gpu.py

[amogkam]

Let's keep this?

[JiahaoYao]

test_ddp.py passed

[JiahaoYao]

can close this pr @krfricke @amogkam