bug: Concurrent chat doesnt work on Mac Silicon

[gabrielle-ong]

Cortex version

1.0.1-203

Describe the Bug

Mac: Concurrent chats for the same model are queued up rather than parallel

• Models tested: tinyllama, llama3.2
• I expect to open 2 CLI windows / Postman window and have concurrent chats
• Works well if separate models (eg tinyllama chat & llama3.2 chat)

May be related to n_parallel parameter in model.yaml

Windows, Ubuntu: Working as expected

Steps to Reproduce

No response

Screenshots / Logs

No response

What is your OS?

• MacOS
• Windows
• Linux

What engine are you running?

• cortex.llamacpp (default)
• cortex.tensorrt-llm (Nvidia GPUs)
• cortex.onnx (NPUs, DirectML)

[gabrielle-ong]

@vansangpfiev do I need to change anything for this to work?
I redownloaded the models, but still is non concurrent on my local com and the VM test-macos-13-1
ie right chat finishes, only then left chat begins

[gabrielle-ong]

works with n_parallel = 2, marking as complete