Validations for 2.2 Release. Cherrry Pick Validation and Manual

[atalman]

TO Validate:

• @atalman illegal memory access for torch.sparse.mm(src, other) / deg.view(-1, 1).clamp_(min=1) pytorch#111574
• @huydhn Fix bug in mem_eff kernel with attention mask and MQA pytorch#116234
• @atalman Back out "[Kineto] Initialize libkineto profilers during torch init process during pybind set-up (#112623)" pytorch#116201 import torch results in cuInit call pytorch#116276
• @atalman Fix missing dependency in torch.utils.tensorboard pytorch#115598
• @huydhn PyTorch conda linux binares should be build with CuSparseLT support pytorch#115085
• @huydhn [DeviceMesh] Rename _device_mesh.py to device_mesh.py to prepare for beta (#115099) pytorch#115193
• @atalman PyTorch is shipped with different versions on NCCL pytorch#112285
• @atalman TensorIteratorBase::is_scalar return false for empty numpy tensors, but true for empty Torch ones pytorch#113037
• @huydhn single-batch torch.bmm is significantly slower with cuBLAS>12.1.0  pytorch#114911
• @atalman Pytorch Home Page does not specify which version of python it requires pytorch#95628
• @huydhn [MPS] Fix memory leak in copy_from_mps_ pytorch#114197

[atalman]

Manual Validations

• CUDA
  • pypi binaries with slimmed dependencies are usable in standard AWS containers (amazonlinux:2 regression in 1.13)
  • pypi binaries with slimmed dependencies are usable with stock Ubuntu-20.04: torch-2.0.0-rc1 and torch-1.13.1 can not be installed on Ubuntu 20.04  pytorch#91067 .
  • PyTorch+CUDA11.8 and CUDA 12.1 has a working FFT on 4090
  • Check cuda 1.12.1 update issue with small wheels, use: torch.linalg.eigh fails on GPU pytorch#94772 (comment)
• torch.compile
  • Basic test works (for example see test mentioned in Search for libdevice relative to shared library triton-lang/triton#1176 ) in PyTorch docker container
  • torch.compile produces a binary which can be used on 3090
  • torch.compile raises an error if used on Windows. Test: https://github.com/pytorch/builder/actions/runs/7505477000/job/20436232769#step:9:611
  • torch.compile works on 3.11 : Test: https://github.com/pytorch/builder/actions/runs/7505477000/job/20436215050#step:11:4844
• MPS
  • Resnet is usable out of the box (i.e. https://github.com/pytorch/builder/blob/main/test/smoke_test/smoke_test.py passes for MPS device). Test: https://github.com/pytorch/builder/actions/runs/7505477000/job/20436239403#step:9:343
• Validate docker release builds

[huydhn]

For pytorch/pytorch#115193 issue with the launching of distributed device mesh API, I follow https://github.com/pytorch/pytorch/blob/main/torch/distributed/_tensor/README.md to run the DTensor example on devgpu torchrun --standalone --nnodes=1 --nproc-per-node=4 dtensor_example.py and it works fine:

$ torchrun --standalone --nnodes=1 --nproc-per-node=4 dtensor_example.py

[2024-01-18 14:08:13,419] torch.distributed.run: [WARNING]
[2024-01-18 14:08:13,419] torch.distributed.run: [WARNING] *****************************************
[2024-01-18 14:08:13,419] torch.distributed.run: [WARNING] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
[2024-01-18 14:08:13,419] torch.distributed.run: [WARNING] *****************************************
NCCL version 2.19.3+cuda12.3
...
DTensor(local_tensor=tensor([[-0.9938,  1.6568, -0.0712,  ..., -0.7047,  0.1956,  0.7011],
        [ 0.0633, -0.0818,  0.0865,  ...,  0.6208, -1.3616,  0.4402],
        [ 0.7410,  0.3713, -1.0218,  ..., -0.6000, -0.3061,  0.0240],
        ...,
        [-0.2041, -0.4914, -1.4949,  ..., -0.6163, -0.6493,  0.5180],
        [ 2.5286, -0.3243,  0.5991,  ...,  0.7855,  0.3508, -0.1411],
        [ 1.6220,  1.5745,  0.4140,  ...,  0.6092, -0.7156,  1.0645]],
       device='cuda:0'), device_mesh=DeviceMesh([0, 1, 2, 3]), placements=(Shard(dim=0),))

For the purpose of doing 2.2.0 release, I think that would be good enough.

[atalman]

Post Release Poetry test:

curl -s https://pypi.org/pypi/torch/2.2.0/json | jq '.info.requires_dist'
[
  "filelock",
  "typing-extensions >=4.8.0",
  "sympy",
  "networkx",
  "jinja2",
  "fsspec",
  "nvidia-cuda-nvrtc-cu12 ==12.1.105 ; platform_system == \"Linux\" and platform_machine == \"x86_64\"",
  "nvidia-cuda-runtime-cu12 ==12.1.105 ; platform_system == \"Linux\" and platform_machine == \"x86_64\"",
  "nvidia-cuda-cupti-cu12 ==12.1.105 ; platform_system == \"Linux\" and platform_machine == \"x86_64\"",
  "nvidia-cudnn-cu12 ==8.9.2.26 ; platform_system == \"Linux\" and platform_machine == \"x86_64\"",
  "nvidia-cublas-cu12 ==12.1.3.1 ; platform_system == \"Linux\" and platform_machine == \"x86_64\"",
  "nvidia-cufft-cu12 ==11.0.2.54 ; platform_system == \"Linux\" and platform_machine == \"x86_64\"",
  "nvidia-curand-cu12 ==10.3.2.106 ; platform_system == \"Linux\" and platform_machine == \"x86_64\"",
  "nvidia-cusolver-cu12 ==11.4.5.107 ; platform_system == \"Linux\" and platform_machine == \"x86_64\"",
  "nvidia-cusparse-cu12 ==12.1.0.106 ; platform_system == \"Linux\" and platform_machine == \"x86_64\"",
  "nvidia-nccl-cu12 ==2.19.3 ; platform_system == \"Linux\" and platform_machine == \"x86_64\"",
  "nvidia-nvtx-cu12 ==12.1.105 ; platform_system == \"Linux\" and platform_machine == \"x86_64\"",
  "triton ==2.2.0 ; platform_system == \"Linux\" and platform_machine == \"x86_64\"",
  "opt-einsum >=3.3 ; extra == 'opt-einsum'",
  "optree >=0.9.1 ; extra == 'optree'"
]