[Question]What does the service parameter max_tokens_in_paged_kv_cache mean?

[wjj19950828]

I have run through the entire process of llama2 and want to stress test and see the benchmark indicators.

Regarding max_tokens_in_paged_kv_cache, I may not understand it well

Is it similar to the max_num_batched_tokens parameter of vllm?

Thanks~

[idealover]

Hi, which GPU did you use?

[wjj19950828]

Hi, which GPU did you use?

A100

[byshiue]

It means the maximun tokens we can save in our paged kv cache.