cuTENSOR error (CUTENSOR_STATUS_NOT_SUPPORTED)

[jvwilliams23]

Hi

As I mentioned on Wednesday, we are getting an CUTENSOR_STATUS_NOT_SUPPORTED when doing lbann.Tessellate (also with lbann.MultiDimReduction mentioned previously #2429) with distconv.

I found that the issue only occurs when mini-batch-size = 1 (or I guess, when mini-batch size < number of gpus). I put the relevant lines from the cuTENSOR log below.

The last entry in the “extent”, which appears to be batch dimension, is split across each gpu (I thought it would be copied). When there is only a single sample, this causes process 0 to have size = 1 in the batch dimension, and process 1 to have size = 0.

[2024-07-12 12:39:08][cuTENSOR][2491675][Api][cutensorInitTensorDescriptor] handle=0X7F84A24F35E0 desc=0X7FFF83FFACE0 numModes=4 extent=[8,8,512,0] stride=[1,8,64,32768] dataType=0 op=1
[2024-07-12 12:39:08][cuTENSOR][2491675][Error][cutensorInitTensorDescriptor] Extent (0) must be > 0.
[2024-07-12 12:39:08][cuTENSOR][2491675][Api][cutensorGetErrorString] error=15
[2024-07-12 12:39:08][cuTENSOR][2491674][Api][cutensorInitTensorDescriptor] handle=0X7FB1558F35E0 desc=0X7FFC7845B6C0 numModes=4 extent=[8,8,512,1] stride=[1,8,64,32768] dataType=0 op=1

Josh