better cache allocation in pytorch engine

[grimoire]

runtime intermediate memory usage might lead to OOM if too many cache blocks are using.
Since most intermediate cache are used by lm_head in ModelForCausalLM, we would leave enough space for the cache and allocate paged cache with rest of the memory.

Tested on V100, prompts with 10k+ input ids.

def main():

    model_path = 'Qwen1.5-7B-Chat'
    backend_config = PytorchEngineConfig(
        session_len=200000,
        cache_max_entry_count=0.99)
    pipe = pipeline(model_path, backend_config=backend_config)

    print('processing...')
    gen_config = GenerationConfig(max_new_tokens=1000)
    result = pipe([prompt], gen_config=gen_config)
    print(result[0].text)

[grimoire]

The estimate is based on vocab_size and max_prefill_token_num
Small max_prefill_token_num would bring a large cache size with the cost of prefill time. we better expose max_prefill_token_num in API.

[lvhan028]

max_prefill_token_num is already defined in PyTorchEngineConfig and TurbomindEngineConfig
What does exposing it in API mean?
Pls define small max_prefill_token_num

[grimoire]

lmdeploy serve api_server can not recognize --max-prefill-token-num

[lvhan028]

是不是不该有默认值？

[grimoire]

是，主要是担心有的模型没有 vocab_size 字段，这样可以保底留一部分 runtime memory。40000x4096x7 ~= 1G

[RunningLeon]

LGTM