seamlessM4T_v2_large finetuning on speech translation task

[laleye]

I'm trying to fintune the seamlessM4T_v2_large model on speech translation task. Would there be a reason for the model to return nan values?

tokens, units = model(batch)
Output

text_logits tensor([[[    nan,     nan,     nan,  ...,     nan,     nan,     nan],
         [ 1.4170, 13.7734,  1.4443,  ...,  1.3828,  1.1865, -1.2373],
         [ 1.4229, 13.6172,  1.4434,  ...,  1.3760,  1.3447, -1.3965],
         ...,
         [    nan,     nan,     nan,  ...,     nan,     nan,     nan],
         [    nan,     nan,     nan,  ...,     nan,     nan,     nan],
         [ 1.4258, 13.3984,  1.4570,  ...,  1.3906,  1.3008, -1.2998]],

        [[    nan,     nan,     nan,  ...,     nan,     nan,     nan],
         [ 1.4121, 13.9219,  1.4453,  ...,  1.3750,  1.3564, -1.4336],
         [ 1.3838, 13.4922,  1.4082,  ...,  1.3525,  1.0928, -1.5068],
         ...,
         [ 1.4268, 13.9609,  1.4561,  ...,  1.3877,  1.2500, -1.6025],
         [ 1.4639, 13.9062,  1.4902,  ...,  1.4316,  1.1572, -1.4883],
         [ 1.4365, 13.1484,  1.4688,  ...,  1.4014,  1.2217, -1.1406]]],
       device='cuda:0', dtype=torch.float16, grad_fn=<UnsafeViewBackward0>)