torch-native pipeline parallelism for big models #3896

Job	Run time
run-tests (latest, test_prod)	48s
run-tests (latest, test_core)	2m 21s
run-tests (latest, test_cli)	2m 59s
run-tests (latest, test_big_modeling)	1m 17s
run-tests (latest, test_deepspeed)	1m 13s
run-tests (latest, test_fsdp)	56s
run-tests (latest, test_example_differences)	1m 21s
run-tests (latest, test_checkpoint_step)	2m 24s
run-tests (latest, test_checkpoint_epoch)	2m 5s
run-tests (latest, test_rest)	6m 7s
run-tests (minimum, test_prod)	48s
run-tests (minimum, test_core)	1m 15s
run-tests (minimum, test_cli)	2m 19s
run-tests (minimum, test_big_modeling)	56s
run-tests (minimum, test_deepspeed)	54s
run-tests (minimum, test_fsdp)	1m 11s
run-tests (minimum, test_example_differences)	52s
run-tests (minimum, test_checkpoint_step)	2m 19s
run-tests (minimum, test_checkpoint_epoch)	2m 11s
run-tests (minimum, test_rest)	6m 4s
	40m 20s

Provide feedback