Fix device mismatch error in GPU decoding with PyTorch 1.13

[pyf98]

Starting from PyTorch 1.13, we will encounter an error if a cpu tensor is indexed by a cuda tensor: https://github.com/pytorch/pytorch/releases/tag/v1.13.0

This affects our standard GPU decoding. (Not sure if it affects other decoding algorithms.)

[codecov]

Codecov Report

Merging #4941 (6bf7987) into master (78c93bc) will decrease coverage by 0.03%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #4941      +/-   ##
==========================================
- Coverage   76.63%   76.60%   -0.03%     
==========================================
  Files         604      604              
  Lines       53934    53934              
==========================================
- Hits        41330    41314      -16     
- Misses      12604    12620      +16     

Flag	Coverage Δ
test_integration_espnet1	66.33% <0.00%> (ø)
test_integration_espnet2	47.42% <100.00%> (+0.01%)	⬆️
test_python	66.45% <100.00%> (-0.10%)	⬇️
test_utils	23.08% <ø> (-0.27%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
espnet/nets/batch_beam_search.py	98.92% <100.00%> (ø)
espnet/nets/pytorch_backend/e2e_asr_maskctc.py	87.40% <0.00%> (-4.45%)	⬇️
espnet2/gan_svs/vits/vits.py	91.61% <0.00%> (-1.80%)	⬇️
espnet2/slu/espnet_model.py	78.48% <0.00%> (-1.75%)	⬇️
espnet2/hubert/espnet_model.py	51.42% <0.00%> (-1.15%)	⬇️
espnet2/gan_tts/vits/vits.py	99.22% <0.00%> (-0.78%)	⬇️
espnet2/diar/espnet_model.py	95.67% <0.00%> (-0.62%)	⬇️
espnet2/svs/espnet_model.py	5.72% <0.00%> (-0.53%)	⬇️
espnet2/bin/asr_transducer_inference.py	93.65% <0.00%> (-0.40%)	⬇️
espnet2/asr/state_spaces/s4.py	52.88% <0.00%> (-0.12%)	⬇️
... and 1 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[sw005320]

Thanks, @pyf98!