On-demand loading via API loads multiple instances of the same model

[jwr]

Which version of LM Studio?
Version 0.4.9+1 (0.4.9+1)

Which operating system?
macOS

What is the bug?
With just-in-time model loading enabled, if several requests come in for the same model within a short period of time, multiple instances of the model will be started. In my specific case, I use a parallelism factor of four, so four instances are loaded, even though only one will be doing actual work on four requests at a time. The "only keep last JIT loaded model" option has no effect.

If I load the model manually, or if I eject the additional instances, things work as expected — until the next time several requests arrive and no model is loaded.

This looks like a race condition.

To Reproduce
Steps to reproduce the behavior:

1. Send several requests in a short period of time when no model is loaded and just-in-time model loading is enabled.
2. Observe several instances of the model being started.

[StateFarmNetwork]

i also experienced this on 0.4.16 build 2

[scips]

I have exactly the same experience and it seems linked to the usage of the same model but with different temperature for instance.

I use the model with CLINE in VSCodium and also I use it in python script in a big loop.
I set the Max Concurrent to 2 but I don't know if it's related or not.
My Model Loading Guardrails is in Relaxed mode
I'm in the runtime: v1.9.1 LM Studio MLX (Apple M5)
Load Anyway is set to: No restriction (Not recommended)

I'm in version: Version 0.4.18+1

See the different temperature between the two versions:

I suppose it's the same if the sampling differs or the output...