sequence parallel default dtype

[stas00]

the newly released nccl finally started to use fp32 accumulation for reduction ops!

• Floating point summation is always done in fp32 accumulators (with the
  exception of fp8 on NVLS, where it uses fp16 inside the switch). Thus,
  the accuracy with fp8 and fp16 data types should be much improved.
  NVIDIA/nccl@72d2432

So we should change the fp32 comms default for SP to the same dtype as inputs if nccl>=2.27.3 - the user can still override the default.

[stas00]

so what's special about a6000 gpu that it fails? I'm looking at the breakage

basically it's this call that fails:

python -c "from torch._dynamo._trace_wrapped_higher_order_op import TransformGetItemToIndex"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: cannot import name 'TransformGetItemToIndex' from 'torch._dynamo._trace_wrapped_higher_order_op' (/usr/local/lib/python3.10/dist-packages/torch/_dynamo/_trace_wrapped_higher_order_op.py)

for older pytorch versions, like pt-2.4, but it succeeds with pt-2.7.1.

Aha! This workflow uses:

torch                         2.6.0a0+df5bbc09d1.nv24.12

so it reports 2.6 but that TransformGetItemToIndex isn't there yet - that's the culprit!

The caller is here:

https://github.com/huggingface/transformers/blob/0725cd6953803b8aacfc85288cbfb83dea30c469/src/transformers/masking_utils.py#L34-L37

Is there a reason why this workflow is locked onto this pytorch version? If so we have to run it on an older transformers version to overcome this issue.

edit: The half-baked pt-2.6 comes from image: nvcr.io/nvidia/pytorch:24.12-py3 - let me try to raise that version.