memory leak when running mistralai/Mistral-7B-Instruct-v0.1

[captify-sivakhno]

I have an error suggesting memory leak when running

sampling_params = SamplingParams(best_of=3, temperature=0.8, top_p=0.95, max_tokens=450, presence_penalty = 1.0, frequency_penalty=1.)
outputs = llm.generate(prompts, sampling_params)

ValueError: Double free! PhysicalTokenBlock(device=Device.GPU, block_number=415, ref_count=0) is already freed.

Any suggestions on what to look into would be most appreciated!

[WoosukKwon]

Hi @captify-sivakhno, thanks for reporting the bug. Could you share your prompts so that we can reproduce the bug?

[captify-sivakhno]

@WoosukKwon please find attached the prompt - it's a simple summary promp
prompt.txt

[frankiedrake]

I have the same issue running inference on multiple GPU

[chadlzx]

The issue seems to occur when the Prompt is too long; I estimate that this threshold of input_id length is above 2048.

[chujiezheng]

@WoosukKwon Hi, is there any progress on this issue?

[hmellor]

Closing in preference to #1556

[manzke]

Watch out - the mistral v0.1 has a sliding window of 4096. If you text is above, it will run into #1556 - either patch the config.json (set "sliding_window": null) or make sure you are below the 4k.