OutOfMemoryError when running gpt2 on T4 GPU

[zhaoyang-star]

I tried running benchmark_throughput.py using hf backend on T4. T4 has 16GB global memory and gpt2 only has 124M params. So there is enough memory space for gpt2. But the error happend as following. Why tried to allocate 8.46 GiB?

Namespace(backend='hf', dataset='/z/Dev/vllm/benchmarks/ShareGPT_V3_unfiltered_cleaned_split.json', hf_max_batch_size=100, model='/z/Models/hf/gpt2', n=1, num_prompts=1000, seed=0, tensor_parallel_size=1, use_beam_search=False)
  0%|                                                                                                                                                                | 0/1000 [00:00<?, ?it/s]The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.
Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.
A decoder-only architecture is being used, but right-padding was detected! For correct generation results, please set `padding_side='left'` when initializing the tokenizer.
Traceback (most recent call last):
  File "benchmark_throughput.py", line 224, in <module>
    main(args)
  File "benchmark_throughput.py", line 187, in main
    elapsed_time = run_hf(requests, args.model, tokenizer, args.n,
  File "benchmark_throughput.py", line 152, in run_hf
    llm_outputs = llm.generate(
  File "/opt/conda/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/transformers/generation/utils.py", line 1572, in generate
    return self.sample(
  File "/opt/conda/lib/python3.8/site-packages/transformers/generation/utils.py", line 2619, in sample
    outputs = self(
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1102, in forward
    lm_logits = self.lm_head(hidden_states)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/linear.py", line 114, in forward
    return F.linear(input, self.weight, self.bias)
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 8.46 GiB (GPU 0; 14.56 GiB total capacity; 5.05 GiB already allocated; 8.25 GiB free; 6.16 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
  0%|