[WIP] Living branch / PR for FSDP development

[voznesenskym]

Stack from ghstack (oldest at bottom):

• Migrate tuple(handle) -> handle #104488
• -> [WIP] Living branch / PR for FSDP development #103711
• SetVariable in dynamo #103205
• [FSDP][6/N] Check valid param freezing for ModuleWrapPolicy #104427
• [FSDP][5/N] Unblock ignored_states + auto wrap (for now) #104418
• [FSDP][4/N] Remove _get_fully_sharded_module_to_states #104409
• [FSDP][3/N] Unify fully_shard auto wrap #104408
• [FSDP][2/N][Easy] Prepare _auto_wrap for fully_shard #104407
• [FSDP][1/N] Move wrapper ModuleWrapPolicy to new path #104346
• [FSDP] Annotate modules for fully_shard #104363

cc @penguinwu @anijain2305 @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @ipiszy @chenyang78

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/103711

• 📄 Preview Python docs built from this PR
• 📄 Preview C++ docs built from this PR
• ❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❌ 46 New Failures, 1 Unrelated Failure

As of commit fef7347:

NEW FAILURES - The following jobs have failed:

• bc_linter (gh)
• cuda11.8-py3.10-gcc7-sm86 / test (inductor_distributed, 1, 1, linux.g5.12xlarge.nvidia.gpu) (gh)
• cuda11.8-py3.10-gcc7-sm86 / test (inductor_huggingface_dynamic, 1, 1, linux.g5.4xlarge.nvidia.gpu) (gh)
• cuda11.8-py3.10-gcc7-sm86 / test (inductor_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) (gh)
• cuda11.8-py3.10-gcc7-sm86 / test (inductor_torchbench_dynamic, 1, 1, linux.g5.4xlarge.nvidia.gpu) (gh)
• cuda11.8-py3.10-gcc7-sm86 / test (inductor, 1, 1, linux.g5.4xlarge.nvidia.gpu) (gh)
• cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (aot_eager_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) (gh)
• cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamic_aot_eager_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) (gh)
• cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamo_eager_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu) (gh)
• lintrunner / linux-job (gh)
• linux-bionic-cuda11.8-py3.10-gcc9 / test (distributed, 1, 3, linux.8xlarge.nvidia.gpu) (gh)
• linux-bionic-cuda11.8-py3.10-gcc9 / test (distributed, 2, 3, linux.8xlarge.nvidia.gpu) (gh)
• linux-bionic-cuda11.8-py3.10-gcc9 / test (distributed, 3, 3, linux.8xlarge.nvidia.gpu) (gh)
• linux-bionic-cuda12.1-py3.10-gcc9 / test (default, 1, 5, linux.4xlarge.nvidia.gpu) (gh)
• linux-bionic-cuda12.1-py3.10-gcc9 / test (default, 2, 5, linux.4xlarge.nvidia.gpu) (gh)
• linux-bionic-cuda12.1-py3.10-gcc9 / test (default, 3, 5, linux.4xlarge.nvidia.gpu) (gh)
• linux-bionic-cuda12.1-py3.10-gcc9 / test (default, 4, 5, linux.4xlarge.nvidia.gpu) (gh)
• linux-bionic-cuda12.1-py3.10-gcc9-bazel-test / build-and-test (default, 1, 1, linux.4xlarge.nvidia.gpu) (gh)
• linux-bionic-cuda12.1-py3.10-gcc9-sm86 / test (default, 1, 5, linux.g5.4xlarge.nvidia.gpu) (gh)
• linux-bionic-cuda12.1-py3.10-gcc9-sm86 / test (default, 2, 5, linux.g5.4xlarge.nvidia.gpu) (gh)
• linux-bionic-cuda12.1-py3.10-gcc9-sm86 / test (default, 3, 5, linux.g5.4xlarge.nvidia.gpu) (gh)
• linux-bionic-cuda12.1-py3.10-gcc9-sm86 / test (default, 4, 5, linux.g5.4xlarge.nvidia.gpu) (gh)
• linux-bionic-py3_8-clang8-xla / test (xla, 1, 1, linux.12xlarge) (gh)
• linux-bionic-py3.11-clang9 / test (crossref, 2, 2, linux.2xlarge) (gh)
• linux-bionic-py3.11-clang9 / test (default, 1, 3, linux.2xlarge) (gh)
• linux-bionic-py3.11-clang9 / test (default, 2, 3, linux.2xlarge) (gh)
• linux-bionic-py3.11-clang9 / test (default, 3, 3, linux.2xlarge) (gh)
• linux-bionic-py3.11-clang9 / test (dynamo, 1, 2, linux.2xlarge) (gh)
• linux-bionic-py3.11-clang9 / test (dynamo, 2, 2, linux.2xlarge) (gh)
• linux-bionic-py3.8-clang9 / test (crossref, 2, 2, linux.2xlarge) (gh)
• linux-bionic-py3.8-clang9 / test (default, 1, 3, linux.2xlarge) (gh)
• linux-bionic-py3.8-clang9 / test (default, 2, 3, linux.2xlarge) (gh)
• linux-bionic-py3.8-clang9 / test (default, 3, 3, linux.2xlarge) (gh)
• linux-bionic-py3.8-clang9 / test (dynamo, 1, 2, linux.2xlarge) (gh)
• linux-bionic-py3.8-clang9 / test (dynamo, 2, 2, linux.2xlarge) (gh)
• linux-focal-cpu-py3.8-gcc7-inductor / test (inductor_huggingface_cpu_accuracy, 1, 1, linux.12xlarge) (gh)
• linux-focal-cpu-py3.8-gcc7-inductor / test (inductor_huggingface_dynamic_cpu_accuracy, 1, 1, linux.12xlarge) (gh)
• linux-focal-py3.8-gcc7 / test (default, 1, 3, linux.2xlarge) (gh)
• linux-focal-py3.8-gcc7 / test (default, 2, 3, linux.2xlarge) (gh)
• linux-focal-py3.8-gcc7 / test (default, 3, 3, linux.2xlarge) (gh)
• linux-focal-py3.8-gcc7 / test (distributed, 2, 2, linux.2xlarge) (gh)
• linux-focal-py3.8-gcc7 / test (docs_test, 1, 1, linux.2xlarge) (gh)
• linux-focal-py3.9-clang7-asan / test (default, 2, 6, linux.4xlarge) (gh)
• linux-focal-py3.9-clang7-asan / test (default, 3, 6, linux.4xlarge) (gh)
• linux-focal-py3.9-clang7-asan / test (default, 4, 6, linux.4xlarge) (gh)
• linux-focal-py3.9-clang7-asan / test (default, 6, 6, linux.4xlarge) (gh)

UNSTABLE - The following job failed but was likely due to flakiness present on trunk and has been marked as unstable:

• cuda11.8-py3.10-gcc7-sm86 / test (inductor_torchbench, 1, 1, linux.g5.4xlarge.nvidia.gpu, unstable) (gh)

This comment was automatically generated by Dr. CI and updates every 15 minutes.