[DTensor] Computed DTensorSpec hash lazily

[awgu]

Stack from ghstack (oldest at bottom):

• -> [DTensor] Computed DTensorSpec hash lazily #114322
• [DTensor] Reduced to one isinstance call in is_shard #114140
• [DTensor] Cached hash for DTensorSpec #113915
• [DTensor] Replaced neg dim normalization with assert in helper #114141
• [DTensor] Used new placements for neg dim in distribute_tensor #113930
• [DTensor] Ensured grad_placements was tuple #113925
• [DTensor] Used new placements for neg dim in from_local #114134
• [DTensor] Used new placements for neg dim in redistribute #113924
• [DTensor] Made _Partial, Replicate frozen dataclasses #113919

This is a forward fix for #113781.

We lazily compute the hash so that we do not try to compute the hash on SymInts (for the stride) during Dynamo tracing.

Tested via:

python test/distributed/_tensor/test_dtensor_compile.py -k test_2d_fsdp_tp_ac_compile

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/114322

• 📄 Preview Python docs built from this PR
• 📄 Preview C++ docs built from this PR
• ❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure

As of commit 0f5e76b with merge base 140c54e ():

NEW FAILURE - The following job has failed:

• periodic / macos-12-py3-x86-64 / build (gh)

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[wanchaol]

sounds good, thanks for the quick fix!

[awgu]

@pytorchbot merge

[pytorchmergebot]

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

[pytorchmergebot]

Merge failed

Reason: 1 jobs have failed, first few of them are: periodic / macos-12-py3-x86-64 / build

Details for Dev Infra team

Raised by workflow job

[awgu]

Failure unrelated: periodic / macos-12-py3-x86-64 / build (gh)

FAILED: caffe2/CMakeFiles/torch_cpu.dir/__/aten/src/ATen/RegisterMPS.cpp.o 
/var/folders/3s/vfzpb5r51gs6y328rmlgzm7c0000gn/T/tmp.4xorXdVd/clang++ -DAT_PER_OPERATOR_HEADERS -DCAFFE2_BUILD_MAIN_LIB -DCPUINFO_SUPPORTED_PLATFORM=1 -DFMT_HEADER_ONLY=1 -DFXDIV_USE_INLINE_ASSEMBLY=0 -DHAVE_MMAP=1 -DHAVE_SHM_OPEN=1 -DHAVE_SHM_UNLINK=1 -DIDEEP_USE_MKL -DMINIZ_DISABLE_ZIP_READER_CRC32_CHECKS -DNNP_CONVOLUTION_ONLY=0 -DNNP_INFERENCE_ONLY=0 -DONNXIFI_ENABLE_EXT=1 -DONNX_ML=1 -DONNX_NAMESPACE=onnx_torch -DUSE_EXTERNAL_MZCRC -D_FILE_OFFSET_BITS=64 -Dtorch_cpu_EXPORTS -I/Users/runner/work/pytorch/pytorch/build/aten/src -I/Users/runner/work/pytorch/pytorch/aten/src -I/Users/runner/work/pytorch/pytorch/build -I/Users/runner/work/pytorch/pytorch -I/Users/runner/work/pytorch/pytorch/cmake/../third_party/benchmark/include -I/Users/runner/work/pytorch/pytorch/third_party/onnx -I/Users/runner/work/pytorch/pytorch/build/third_party/onnx -I/Users/runner/work/pytorch/pytorch/third_party/foxi -I/Users/runner/work/pytorch/pytorch/build/third_party/foxi -I/Users/runner/work/pytorch/pytorch/torch/csrc/api -I/Users/runner/work/pytorch/pytorch/torch/csrc/api/include -I/Users/runner/work/pytorch/pytorch/caffe2/aten/src/TH -I/Users/runner/work/pytorch/pytorch/build/caffe2/aten/src/TH -I/Users/runner/work/pytorch/pytorch/build/caffe2/aten/src -I/Users/runner/work/pytorch/pytorch/build/caffe2/../aten/src -I/Users/runner/work/pytorch/pytorch/torch/csrc -I/Users/runner/work/pytorch/pytorch/third_party/miniz-2.1.0 -I/Users/runner/work/pytorch/pytorch/third_party/kineto/libkineto/include -I/Users/runner/work/pytorch/pytorch/third_party/kineto/libkineto/src -I/Users/runner/work/pytorch/pytorch/aten/src/ATen/.. -I/Users/runner/work/pytorch/pytorch/third_party/FXdiv/include -I/Users/runner/work/pytorch/pytorch/c10/.. -I/Users/runner/work/pytorch/pytorch/third_party/pthreadpool/include -I/Users/runner/work/pytorch/pytorch/third_party/cpuinfo/include -I/Users/runner/work/pytorch/pytorch/third_party/QNNPACK/include -I/Users/runner/work/pytorch/pytorch/aten/src/ATen/native/quantized/cpu/qnnpack/include -I/Users/runner/work/pytorch/pytorch/aten/src/ATen/native/quantized/cpu/qnnpack/src -I/Users/runner/work/pytorch/pytorch/third_party/cpuinfo/deps/clog/include -I/Users/runner/work/pytorch/pytorch/third_party/NNPACK/include -I/Users/runner/work/pytorch/pytorch/third_party/fbgemm/include -I/Users/runner/work/pytorch/pytorch/third_party/fbgemm -I/Users/runner/work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit/src -I/Users/runner/work/pytorch/pytorch/third_party/ittapi/src/ittnotify -I/Users/runner/work/pytorch/pytorch/third_party/FP16/include -I/Users/runner/work/pytorch/pytorch/third_party/fmt/include -I/Users/runner/work/pytorch/pytorch/build/third_party/ideep/mkl-dnn/include -I/Users/runner/work/pytorch/pytorch/third_party/ideep/mkl-dnn/src/../include -I/Users/runner/work/pytorch/pytorch/third_party/flatbuffers/include -isystem /Users/runner/work/pytorch/pytorch/cmake/../third_party/googletest/googlemock/include -isystem /Users/runner/work/pytorch/pytorch/cmake/../third_party/googletest/googletest/include -isystem /Users/runner/work/pytorch/pytorch/third_party/protobuf/src -isystem /Users/runner/work/_temp/conda_environment_6951118264/include -isystem /Users/runner/work/pytorch/pytorch/third_party/gemmlowp -isystem /Users/runner/work/pytorch/pytorch/third_party/neon2sse -isystem /Users/runner/work/pytorch/pytorch/third_party/XNNPACK/include -isystem /Users/runner/work/pytorch/pytorch/third_party/ittapi/include -isystem /Users/runner/work/pytorch/pytorch/cmake/../third_party/eigen -isystem /Users/runner/work/pytorch/pytorch/third_party/ideep/mkl-dnn/include/oneapi/dnnl -isystem /Users/runner/work/pytorch/pytorch/third_party/ideep/include -isystem /Users/runner/work/pytorch/pytorch/build/include -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=braced-scalar-init -Werror=range-loop-construct -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wvla-extension -Wsuggest-override -Wnewline-eof -Winconsistent-missing-override -Winconsistent-missing-destructor-override -Wno-pass-failed -Wno-error=pedantic -Wno-error=old-style-cast -Wno-error=inconsistent-missing-override -Wno-error=inconsistent-missing-destructor-override -Wconstant-conversion -Wno-invalid-partial-specialization -Wno-missing-braces -Qunused-arguments -fcolor-diagnostics -faligned-new -Werror -Wno-unused-but-set-variable -fno-math-errno -fno-trapping-math -Werror=format -DUSE_MPS -Wno-unused-private-field -Wno-missing-braces -DHAVE_AVX2_CPU_DEFINITION -O3 -DNDEBUG -DNDEBUG -isysroot /Applications/Xcode_13.3.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.3.sdk -mmacosx-version-min=11.0 -fPIC -DMKL_HAS_SBGEMM -Wall -Wextra -Wdeprecated -Wno-unused-parameter -Wno-unused-function -Wno-missing-field-initializers -Wno-unknown-pragmas -Wno-type-limits -Wno-array-bounds -Wno-strict-overflow -Wno-strict-aliasing -fvisibility=hidden -O2 -Wmissing-prototypes -Werror=missing-prototypes -DASMJIT_STATIC -Xpreprocessor -fopenmp -I/Users/runner/work/_temp/conda_environment_6951118264/include -std=gnu++17 -Wno-missing-prototypes -Wno-error=missing-prototypes -MD -MT caffe2/CMakeFiles/torch_cpu.dir/__/aten/src/ATen/RegisterMPS.cpp.o -MF caffe2/CMakeFiles/torch_cpu.dir/__/aten/src/ATen/RegisterMPS.cpp.o.d -o caffe2/CMakeFiles/torch_cpu.dir/__/aten/src/ATen/RegisterMPS.cpp.o -c /Users/runner/work/pytorch/pytorch/build/aten/src/ATen/RegisterMPS.cpp
clang: error: unable to execute command: Segmentation fault: 11
clang: error: clang frontend command failed due to signal (use -v to see invocation)
Apple clang version 13.1.6 (clang-1316.0.21.2.3)
Target: x86_64-apple-darwin21.6.0
Thread model: posix
InstalledDir: /Applications/Xcode_13.3.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
clang: note: diagnostic msg: 
********************

[awgu]

@pytorchbot merge -i

[pytorchmergebot]

Merge started

Your change will be merged while ignoring the following 1 checks: periodic / macos-12-py3-x86-64 / build

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

[huydhn]

This is weird as I still see the test failing in trunk on multigpu job https://hud.pytorch.org/pytorch/pytorch/commit/e7326ec295559c16795088e79a5631e784bb4d61, any thoughts?

[awgu]

Looking into it!