Hi,
Even with the patch ( #486 ) we only have 1 slot avalable for concurrency. Is there any plan to introduce multi-slot for FLM ? This will permit to have multi-task with different prompt memories, using the huge prefill speed that you provide.
Thank you for your hard work. Happy to see that you joined AMD?
Hi,
Even with the patch ( #486 ) we only have 1 slot avalable for concurrency. Is there any plan to introduce multi-slot for FLM ? This will permit to have multi-task with different prompt memories, using the huge prefill speed that you provide.
Thank you for your hard work. Happy to see that you joined AMD?