[T5] Fix speed degradation bug t5

[patrickvonplaten]

What does this PR do?

Checking every value of a tensor for inf is expensive. This was added to T5 to allow for fp16 training, but should then also be used when the model is in fp16 to not slow down normal fp32 mode.

Using @dsgissin script:

device = torch.device('cuda:0') if torch.cuda.is_available() else torch.device('cpu')
print(f"Using device: {device}")

t5_tokenizer = T5TokenizerFast.from_pretrained('t5-base')
t5_model = T5ForConditionalGeneration.from_pretrained('t5-base')
t5_model = t5_model.to(device)

t5_input_ids = t5_tokenizer("summarize: studies have shown that owning a dog is good for you ", return_tensors="pt").input_ids  # Batch size 1
t5_input_ids = t5_input_ids.to(device)

import time
import numpy as np
N = 100
times = []
for _ in range(N):
  start = time.time()
  t5_outputs = t5_model.generate(t5_input_ids)
  end = time.time()
  times.append(end-start)
print(f"transformers version: {transformers_version}")
print(f"torch version: {torch_version}")
print(f"{1000*np.mean(times):.0f} ms \u00B1 {1000*np.std(times):.2f} ms per loop (mean \u00B1 std of {N} runs)")

with:

• Python 3.8.5
• PyTorch 1.7.1
• CUDA 11.1 on a NVIDIA V100 GPU

The time was improved from:
441 ms ± 41.67 ms per loop (mean ± std of 100 runs)
to
388 ms ± 44.75 ms per loop (mean ± std of 100 runs)

[patil-suraj]

Looks good to me!

Some of the other library models also use this trick (BART-like models), we should also investigate those.

[patrickvonplaten]

Looks good to me!

Some of the other library models also use this trick (BART-like models), we should also investigate those.

Good point - yeah, let me fix this in this PR actually