how to slove indexSelectSmallIndex: block: [1,0,0], thread: [32,0,0] Assertion srcIndex < srcSelectDimSize failed when training ASILLE3

[foolishcqx]

raceback (most recent call last): | 6/1367 [00:06<23:50, 1.05s/it]
File "/home/zhongjiafeng/SS/chenqixian/TTS/Fastspeech2/train.py", line 199, in
main(args, configs)
File "/home/zhongjiafeng/SS/chenqixian/TTS/Fastspeech2/train.py", line 82, in main
output = model(*(batch[2:]))
File "/home/zhongjiafeng/anaconda3/envs/d2l/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/home/zhongjiafeng/SS/chenqixian/TTS/Fastspeech2/model/fastspeech2.py", line 81, in forward
) = self.variance_adaptor(
File "/home/zhongjiafeng/anaconda3/envs/d2l/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/home/zhongjiafeng/SS/chenqixian/TTS/Fastspeech2/model/modules.py", line 117, in forward
pitch_prediction, pitch_embedding = self.get_pitch_embedding(
File "/home/zhongjiafeng/SS/chenqixian/TTS/Fastspeech2/model/modules.py", line 80, in get_pitch_embedding
prediction = self.pitch_predictor(x, mask)
File "/home/zhongjiafeng/anaconda3/envs/d2l/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/home/zhongjiafeng/SS/chenqixian/TTS/Fastspeech2/model/modules.py", line 247, in forward
out = out.masked_fill(mask, 0.0)
RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
../aten/src/ATen/native/cuda/Indexing.cu:922: indexSelectSmallIndex: block: [1,0,0], thread: [32,0,0] Assertion srcIndex < srcSelectDimSize failed.
../aten/src/ATen/native/cuda/Indexing.cu:922: indexSelectSmallIndex: block: [1,0,0], thread: [33,0,0] Assertion srcIndex < srcSelectDimSize failed.

[decajcd]

Have you reolved it?