You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I just did a quick implementation of gpt-fast and did an inference on the Llama-2 7B. I seem to get around 65 tokens per second on average without quantization.
I also did a quick inference on the normal implementation of the model and I seem to get around 150 tokens per second on average.(My calc: Total no of tokens/total time taken)
Is there anything I am missing.
My hardware spec:
GPU: NVIDIA GeForce RTX 4090
Cpu cores: 32
Is the "no of CPU cores" a hyper parameter in gpt-fast? Is there a threshold for the "no of CPU cores", below which only the CPU overhead occurs and gpt-fast helps only in such cases?
Please correct me If I am wrong.
Thanks in advance.
The text was updated successfully, but these errors were encountered:
To be clear, the metric reported here is also sometimes called "tokens per second per user" (i.e. the latency for a single request). So the benchmarks here are all run with BS=1. I think the number you're getting (65 tokens per second) is pretty close to the peak you can get on your hardware (since 65 * 7 * 2 = 910 GB/s, while the 4090 has 1 TB/s of memory bandwidth).
If you're running with a larger batch size you can easily obtain more "tokens/s".
Hi,
I just did a quick implementation of gpt-fast and did an inference on the Llama-2 7B. I seem to get around 65 tokens per second on average without quantization.
I also did a quick inference on the normal implementation of the model and I seem to get around 150 tokens per second on average.(My calc: Total no of tokens/total time taken)
Is there anything I am missing.
My hardware spec:
GPU: NVIDIA GeForce RTX 4090
Cpu cores: 32
Is the "no of CPU cores" a hyper parameter in gpt-fast? Is there a threshold for the "no of CPU cores", below which only the CPU overhead occurs and gpt-fast helps only in such cases?
Please correct me If I am wrong.
Thanks in advance.
The text was updated successfully, but these errors were encountered: